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ABSTRACT 

This two-part document reports the results of a 
literature review of workplace literacy program evaluation and then 
details a study of two current workplace literacy programs. The five 
chapters of the first part summarize findings on what is known about 
workplace literacy programs, methods for evaluating workplace 
literacy programs, assessing workplace literacy program results, 
assessing the impact on family literacy, and assessing the impact on 
productivity. The second part describes the study, the purposes of 
which were to develop an impact assessment model for workplace 
literacy programs and to produce data on the impact of programs at 
two sites, as well as to refine the model for use at other sites. The 
two sites chosen were very different but both operate established 
programs involving technical and communications training, high school 
equivalency programs, and English-as-a-Second-Language classes. Pre- 
and post-program data were gathered on learners 1 job productivity, 
literacy attributes, and literacy practices with their families 
through interviews, tests, and questionnaires, and supervisor 
ratings. Results showad that learners made gains in their literacy 
self-image, their ability to articulate plans, the amount and range 
of literacy activity both at work and away from work, and their 
reading strategies and comprehension. The study concluded that the 
evaluation process developed can be used as a model for evaluation of 
workplace evaluation projects. Eight appendices contain project 
forms, and a bibliography lists 79 references. (KC) 



00 
CO 



EVALUATING THE IMPACT 
OF WORKPLACE LITERACY PROGRAMS 

Results and Instruments from the NCAL Workplace Literacy Impact Project 



Larry Mikulecky and Paul Lloyd 

School of Education 
Indiana University - Bloomington 



S3 

v> 



U.t. DEPARTMENT Of EDUCATION 
Office of Educational Research and improvement 

EDUCATIONAL RESOURCES INFORMATION 
y CENTER (ERIC) 

/O/nis document hit been reproduced as 
<S received from the peraon or organisation 
originating it 

□ Minor changes have been made to improve 
reproduction quality 

e Points ot view or opmiona stated in this docu- 
ment do not necessarily represent official 
OERl position or policy 



BEST COPY AVAILABLE 



90'- 2 



Evaluating the Impact of Workplace Literacy 

Programs 

Results and Instruments from the NCAL Workplace Literacy Impact Project 



Larry Mikulecky & Paul Lloyd 

School of Education 
Indiana University-Bloomington 



A research project funded by the 
National Center on Adult Literacy 
University of Pennsylvania 
Philadelphia, Pennsylvania 

July, 1992 



Acknowledgements 



We would like to thank the following people for their assistance in 
connection with this project in all its aspects: developing instruments, 
conducting interviews, arranging site visits, collecting data, analysing 
data, and gathering research material. 

Cumberland Hardwoods : 

Janet Davis, Yvonne Fournier, Keith Girdley, 
John Keisling, Mary Stevens, Mary Ruth Winford. 

Delco Chassis : 

Russell Ater, Karin Lotz. 

Finger Lakes R.E.C.E.D. : 

David Mathes, Daniel O'Connell. 

Indiana University : 

Kathy Bussert, Julie Chen, Mei-Li Chen, Parsa Choudhury, 
Ming-Fen Li, Michele Peers, Sharon Sperry, Zhang Hong. 



University nf Rochester : 
Robert Ferrell 



Contents 



Acknowledgements iii 
Contents v 



9 



Part I 


Research Background on Workplace Literacy 




^napter i 


\X7Vio + Tiro \rn rwir ohni it" Ti7nT"lrT"\l Q r*f* li'fPfQPV TYTOCfrPTTlQ 
W IldL WC IvIlUW dUUUl WU1 ivjJlcli^C 11 LCI ci^y pi u^i oiiio 


3 


Chapter 2 


Methods for evaluating workplace literacy programs 


15 


Chapter 3 


Assessing workplace literacy program results 


25 


Chapter 4 


Assessing impact on family literacy 


39 


Chapter 5 


Assessing impact on productivity 


47 


Part n 


THE CURRENT STUDY 




Chapter 6 


Structure of current study 


61 


Chapter 7 


Results of current study 


75 


Chapter 8 


Discussion and implications 


93 


Bibliography 




109 


Appendix A 


Interview form and instructions 


117 


Appendix B 


Questionnaire form and instructions 


133 


Appendix C 


Cloze test samples and instructions 


143 


Appendix D 


Family literacy focus group interview 


151 


Appendix E 


Classroom observation form 


157 


Appendix F 


ESL checklist 


161 


Appendix G 


Supervisor rating scale examples and instructions 


167 


Appendix H 


Tabular data 


175 









Part I 



RESEARCH BACKGROUND ON 
WORKPLACE LITERACY 



9 

ERIC 



00r 0 - 



Chapter l 

What We Know 
About Workplace Literacy Programs 

Overview 

Though a growing body of research has identified principles and 
elements associated with effective workplace literacy programs, few 
programs are able to incorporate all elements. Evaluation of 
workplace literacy programs is further complicated by the fact that 
there appear to be a variety of workplace literacy problems, each 
calling for a different sort of instruction. Still, over the last two 
decades, we have learned a good deal about what to look for in 
effective workplace literacy programs. 

For example we have learned that: 

• there are several different workplace literacy problems 
which call for a multi-stranded approach to instruction; 

• improvement takes a significant amount of learner 
practice time; 

• transfer from learning one application to new 
applications is very limited; 

• significant learning loss occurs within a few weeks if 
skills are no t practiced. 

We have also learned that effective workplace literacy programs are 
characterized by active involvement of project partners (including 
employees) in systematically determining local literacy needs and 
developing programs. 

Multiple Strands for Multiple Problems 

It is important to realize that we face several literacy problems 
in the workplace and not just one. The person who can't read at all 
requires different support than does the high school graduate who 
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cant meet the new reading demands of his job. People educated in a 
foreign language and not speaking much English require another sort 
of support. Providing the same services and programs to such 
different clients makes no sense, and yet it sometimes occurs. 

Increasingly, programs in business and industry are becoming 
multi-strand programs. In such programs, one instructional strand 
might be available to English as Second Language learners while other 
strands are available for learners wishing to pursue GED certificates in 
preparation for further education or for high school graduates 
preparing for technical training. Even the format for instruction may 
vary from structured classes to small group instruction to computer 
guided learning, to individual tutoring. 

Bussert (1991) surveyed 107 workplace literacy programs 
described in the research literature. Of the descriptions of workplace 
literacy programs providing sufficient in^ttmation for judgments to be 
made, the vast majority (74%) offered a multiple strand curriculum 
(i.e. two or more of the following: ABE, GED, ESL, a selection of basic 
skills /technical courses) while 13% reported self-pacing of learning 
(i.e. home study, PLATO computerized learning, learning modules). 

Improvement Takes Significant Learner Practice Time 

Training material and technical reading material in the 

A 

workplace tends to range in difficulty from upper high school to 
beginning college difficulty levels (Sticht, 1975; Mikulecky, 1982; 
Rush, Moe & Storlie, 1986). Some learners, like high school 
graduates who need to brush-up reading skills, can learn to 
comprehend technical materials with a minimum of instruction time 
(about 30-50 hours). Other learners who have extreme difficulty with 
even simple reading, such as signs or simple sentences, may require 
several hundred hours of instruction or, indeed, may never be able to 
comprehend some technical material. Gains do not come quickly. 
The average program takes approximately 100-120 hours of practice 
time for learners to make the equivalent of a year gain in reading 



WHAT WE KNOW ABOUT WORKPLACE LITERACY PROGRAMS 



ability. Auspos et al (1989) reports that several hundred learners in a 
pre-work literacy program in 13 diverse sites across the country 
averaged 132 hours of basic education. Of the participants tested for 
reading gains using the Test of Adu lt Basic Education, an average .7 of 
a year gain in reading ability after approximately 100 hours of 
instruction was demonstrated. 

Targeted programs using materials which learners encounter 
during everyday activities appear to make more rapid gains, but still 
take from 50-60 hours per grade level gain (Mikulecky, 1989). Sticht 
(1982) reports that military enlisted men receiving 120 hours of 
general reading instruction averaged an improvement of .7 grade 
levels in reading ability. Enlisted men being trained with workplace 
materials improved 2.1 grade levels when reading work related 
materials during the same amount of time. 

Computer learning programs may also cut learning time slightly, 
probably since there is more reading practice and less discussion. 
Haigler (1990) indicates that learners gained an average of 1.26 years 
of reading ability in an average of 78 hours of practice using 
computerized lessons in the JSEP job related basic skills program. 
This is equivalent to about 63 hours of practice for a year of gain. 

Yet, linking learning gain to practice time can be somewhat 
deceptive and misleading. A sense of perspective is needed. A gain of 
one year of reading growth in one hundred twenty hours of practice is 
a bargain compared to the experience of the average school child who 
spends over a thousand hours for a reading gain of one year. 
Furthermore, the more effective workplace literacy programs report 
reducing learning time to 50-70 hours of practice for a year of gain. No 
program, however, has been able to consistently improve reading 
ability from low-level to high school or college standards in 20, 30 or 
even 50 hows. This is important to note because in many industries 
the standard training class is less than thirty hours. 

The fact that literacy gains usually take more time than is 
typically allocated in workplace training programs presents a problem. 
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For gains to occur, more practice time must be found. Effective 
programs demonstrate at least three possibilities for increasing 
practice time. Some programs immerse employees in integrated 
technical/basic skills classes full-time for several weeks (see Delco 
description in Chapter 6). Other programs provide sequences of 
courses allowing learners to move from one course to another and 
eventually to continue learning at technical schools and community 
colleges. A third program type uses workplace materials in training 
classes, and thus reaps the bonus of additional practice time as 
learners read these same materials on the job. 

Transfer to New Applications is Severely Limited 

Research indicates that there is a severe limitation on how much 
literacy will transfer from one type of task to other types of tasks, if 
the new tasks are not part of the training. Reading the Bible is 
considerably different from reading the newspaper which, in turn, 
differs significantly from the sort of thinking one does while reading a 
manual. Perkins and Salomon (1989:19), after reviewing the cognitive 
research from the late 70's through the '80 f s concluded that: 

'To the extent that transfer does take place, it is highly 
specific and must be cued, primed, and guided; it seldom 
occurs spontaneously. The case for generalizable, context- 
independent skills and strategies that can be trained in one 
context and transferred to other domains has proven to be 
more a matter of wishful thinking than hard empirical 
evidence." 

Consistently during the past decade, literacy researchers have 
reminded us that literacy is not something you either do or do not 
have. It is not even a continuum. What we mean by "literacy" is more 
accurately described as "literacies." There is some degree of overlap 
between being able to read one sort of material and being able to read 
other sorts. The degree of overlap between reading a short story, a 
poem, a lab manual, an equation, a computer screen, a census report, 
and government documents may be severely limited, however. We 
know very little about the degree of overlap and the degree of 
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difference among these various literacy formats and tasks* 

Evidence suggesting the limitations of transferring literacy skills 
is found in the results from the National Assessment of Educational 
Progress, which surveyed the literacy skills of young adults (Kirsch & 
Jungeblutt, 1986)- This survey measured the literacy abilities of young 
adults in three different areas: their use of prose, document, and 
quantitative forms of literacy. Correlations among subject 
performances in these three areas revealed limited overlap in literacy 
abilities (i*e. about 25% shared variance)* Among those surveyed, 
being able to read a newspaper was only partially related to being able 
to make sense of a document like a chart, table, or form* Some 
literacy ability apparently will transfer* Document reading and prose 
reading, for example, did not seem to be totally separated skills* For 
most learners, however, this degree of shared literacy ability appears 
to be Insufficient for transfer to occur easily. The idea of teaching 
someone to read a poem and expecting that s/he is going to improve 
also in ability to read a computer screen is probably a misplaced hope* 
What we want people to be able to do, we need to teach them* Some 
people are able to make great transfers from one situation to others* 
Such people, unfortunately, do not appear to be the norm. 

The limitations of literacy transfer have serious implications for 
workplace literacy programs* This is especially true if programs 
attempt to use traditional, school-type materials. Sticht (1982) found 
that general literacy training did not transfer to job applications* He 
now recommends a "functional context" approach which teaches 
literacy by using the materials with which one is likely to function on a 
daily basis* 

Significant Learning Loss Occurs without Regular Practice 

The problem of lack of transfer is related to the problem of 
learning loss. When a person cannot use what they have learned in 
real-world situations, they tend to lose the new skill because they lose 
the chance to practice it. This is important, because new knowledge 
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must be used or it is lost, Sticht's (1982) report of military studies 
indicates that enlisted men improved in literacy abilities while they 
were in general literacy classes, but that within eight weeks, 80% of 
those gains were lost. The only exception to this finding occurred 
when job-related materials were used to teach literacy abilities. In the 
latter case, learning gains held up, probably because learners were able 
to see transfer and continued to practice the abilities thej^ had 
mastered. 

This finding is very important. It means that efforts and 
resources can be squandered if learners are taught with general 
materials which have no relationship to the materials they see daily. It 
also suggests that the timing of workplace literacy training is 
important. Preparing learners for the basic skills which are 
demanded by new jobs may be wasted if learners must wait several 
months before they are able to apply and practice their new learning. 

Some programs (Mikulecky & Philippi, 1990) have analyzed 
specific job tasks and developed instructional materials using both 
work and everyday materials. For example, in banking, the careful 
reading of withdrawal and deposit slips involves reading, computation, 
and judgment. Similar skills are requLed at home in reading and 
filling out forms for mail-order catalogs and in paying some bills. 
Instruction that alternates applying the same strategies to workplace 
and home materials offers an increased possibility for practice. Data is 
not yet available on the effectiveness of this strategy in stemming 
learning loss. 

Effective Workplace Literacy Programs 

To be effective, therefore, workplace literacy programs must 
have well designed instruction and they must be flexible enough to 
meet both the needs of differing learners and of changing situations on 
the job. The discussion so far has highlighted the importance of 
designing programs which integrate workplace basic skills instruction 
with several other types of instruction, for example, technical training, 
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E.S.L. training, G.E.D. instruction, and low level literacy training. It 
has emphasized the importance of countering lack of transfer and 
learning toss by providing long-term practice with materials and 
activities directly related to the learners' everyday demands. 
Additional elements of effective programs are apparent as we examine 
more workplace literacy training across the country. 

Salient Features of Effectiv e Programs 

A recent study of 37 workplace literacy programs funded by the 
U. S. federal government (Kutner et al, 1991) identified four key 
components of effective programs. One of these elements related to 
directly linking instructional materials to literacy tasks identified 
during job analyses. This connection was discussed at some length 
earlier in this chapter. In addition to this clear link between 
instruction and job tasks, effective programs were characterized by: 

1) active involvement by project partners; 

2) active involvement by employees in determining literacy 
needs; 

3) systematic analysis of on-the-job literacy requirements. 

Bussert (1991) reported most workplace literacy programs 
involve partnerships of some sort. Bussert analyzed descriptions of 
107 U.S. workplace literacy programs and found 92% to involve 2 or 
more partners. Sometimes the partners were multiple unions or 
multiple businesses; or a school and business; or a government agency, 
a business, and a union. The most common types of partnerships 
among the programs she surveyed were the following: 

• Employers working with others 88% 

• Schools (public school, community college, and 51% 
university) working in partnership with others 

• Unions working with others 34%. 

Recruitment and retention were reported to be effective when 
each partner played an active role during the early stages of program 
development and a continuing role in supporting program goals. 
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Involvement went beyond leaders, however, to include learners 
themselves, who helped gather materials and made suggestions for 
expanding the collection of custom-designed materials. It was usually 
those closest to the job who knew what strategies would be most 
effective in gathering information and solving job problems. Active 
participation of partners sometimes meant supervisors and top job 
performers helping to analyze job tasks and suggesting materials and 
approaches which they found effective in preparing new workers. 

A few specific examples of effective workplace programs can 
help illustrate these elements of effective programs. Earlier 
references to the military programs described by Sticht (1982) and to 
the computerized JSEP program described by Halgler (1990) touched 
on these elements. Now we will examine examples provided by 
Hargroves (1989) and Mikulecky & Strange (1986). 

Boston Federal Reserve Bank's Skills Development Center 
Hargroves (1989) describes a well-established workplace basic 
skills program in the banking industry. She presents the results of a 
15 year comparison of the Federal Reserve Bank's basic skills trainees 
to a peer group of entry level workers in terms of: 1) the effectiveness 
of training in helping under-educated youth catch up, 2) retention, 
3) job performance and 4) earning power. The Bank's skills 
development program integrated basic skills with clerical training, 
supervised work experience, and counseling. Trainees came into the 
program because they lacked basic skills which were needed in most 
clerical jobs. Though 50% of the trainees had graduated from high 
school, half read at or below the eighth grade level. Two out of three 
Skills trainees attended long enough to complete an extensive class 
and on-the-job training program leading to job placement at the Bank. 

Hargroves (1989) gathered information on 207 Skills Center 
trainees from 1973 to 1988 and compared employment data to that of 
301 Bank employees hired for entry-level positions from 1974 to 
1986. Her results indicated that several months of formal training 
combined with on-the-job experience and counseling enabled under- 
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educated youth to catch up to typical entry-level workers. Two thirds 
of the trainees (who would not otherwise have been eligible for 
employment) were placed in jobs. The trainees, on the average, 
stayed longer than their entry-level peers, despite the fact that in the 
late 1980's there was a low unemployment rate and ample job 
opportunities outside the bank. The majority of Skills Center 
graduates earned as much as their entry-level peers who were both 
more educated and more experienced. "In summary, the program 
produced a supply of employees who were trained as well or better 
than other new entry-level employees and understood the Bank's 
employment practices; it also provided trainees to departments on 
short notice for extra clerical help" (Hargroves, 1989: 67). 

Several elements key to program success were highlighted by 
the Hargroves study. These include: 1) integrating basic skills, clerical 
skills, work experience and intensive counseling, 2) self-paced and 
often one-on-one instruction focusing on competence, 3) connections 
to community agencies for recruitment, and 4) good communications 
with Bank supervisors in order to develop job placements. 

Two Long-Term Integrated Skill Programs 

Mikulecky and Strange (1986) reported on a program to train 
word processor operators and a second program to train wastewater 
treatment workers. Each program involved extensive training time. 
The word processor operators were paid to attend between 14 and 20 
weeks of training, 40 hours per week. The number of weeks was 
determined by their ability to function at levels comparable to those of 
average word processor operators who were currently employed. The 
training program for the wastewater treatment plant involved 20 full 
weeks of voluntary training which alternated classroom training with 
on-the-job training. The word processor training program screened 
applicants and accepted no applicants who read more than 3 grade 
levels below the difficulty level of the business materials read, typed 
and edited by existing word processor operators. These materials 
ranged from high school to college level in difficulty. The wastewater 
treatment training program, on the other hand, provided 

ir 13 
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approximately 100 hours of special literacy support for the least 
academically able of its workers. This support focused on preparing 
the use of job and training materials which averaged from 11th grade 
to college level in difficulty. Employers and top-performing workers 
helped to analyze job tasks and provide benchmarks for acceptable 
performance. 

The average learner in the word processor training program 
reached job-level competence in 20 weeks. Some of the trainees were 
able to find employment in 14 weeks, while a few took nearly 28 
weeks. The program concluded in the middle of a recession, during 
which 1/3 of the cooperating companies stopped all hiring. In spite of 
these economic difficulties, 70 percent of the program participants 
found employment as word processors within two months of 
completing the program. 

The wastewater treatment program focused on the least literate 
20 percent of its workers. Nearly 1/2 passed their technical training 
post tests. The consensus of technical instructors was that less than 5 
percent would have passed without the support of the literacy 
program. Of students attending special training sessions, nearly 70 
percent were able to summarize job materials in their own words by 
the end of training. Only about 10 percent of the learners 
demonstrated gains in general reading abilities and those were 
students who invested 5 or more hours weekly outside of class on 
general reading materials. Retention of students receiving special 
basic skills training was higher than that of more able students who 
attended technical training only. 

Conclusion 

No single class or course seems able to meet the demands of the 
diverse populations within a workplace nor to provide a sufficient 
amount of instruction to move very low-level literates to the functional 
literacy levels called for in today's workplace. Multi-strand 
approaches, which involve several different types of courses and 
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strings of educational experiences leading to long-term training goals, 
appear to offer the highest probability of success. Such programs need 
to encourage learners to practice and retain new skills by linking 
training materials to job and home literacy demands. The active 
involvement of workplace partners appears to be key to establishing 
those links and to systematically analyzing program effectiveness. 
Relatively few workplace literacy programs meet all these 
effectiveness criteria, but the degree to which these criteria can be 
accommodated appears directly related to program success. 
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Chapter 2 



Methods for Evaluating 
Workplace Literacy Programs 



Overview 

The previous chapter presented examples of effective workplace 
literacy programs and identified key program parameters. To evaluate 
workplace literacy programs effectively, two types of evaluation are 
desirable ~ formative evaluation and summative evaluation. 

Formative evaluation of a workplace literacy program takes place 
during beginning and middle stages of program operation. The 
purpose is to identify problem areas which can be addressed and 
modified while change is still possible and productive. This type of 
evaluation usually involves the use of interviews, document analysis, 
and observations to determine: 

• the degree to which all involved with the program 
understand and share program goals; 

• whether the resources in terms of personnel, 
materials, learning environment, and learner time 
are sufficient, given current knowledge, to achieve 
the goals; 

• whether the learning processes and methods 
employed appear to be sufficient to accomplish the 
goals. 

Summative evaluation of workplace literacy programs usually 
takes place at the end of program operation and is designed to assess 
how well the program has succeeded. This sort of evaluation requires 
gathering pre and post program data and then analyzing that data. 
This implies using and developing measures directly related to 
program goals. Typical goals for workplace literacy programs include 
improved learner literacy abilities, improved literacy practices at work 
and elsewhere, changed learner beliefs about literacy, self, and 
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education, and improved learner productivity on the job- Assessment 
is often accomplished through use of formal standardized tests, 
informally constructed tests related to the workplace, questionnaires 
related to literacy practices, and interviews with learners and 
supervisors- In addition, company records and ratings on productivity, 
safety, attendance, and enrollment in subsequent classes can expand 
the evidence available for assessing program impact. 

Current Workplace Literacy Program Evaluations 

Only a few workplace literacy programs described in the 
research literature report rigorous program evaluations or careful 
documentation of learner gains, impacts on productivity, and detailed 
descriptions of effective program practices. Some of these programs 
have been cited in the previous chapter. See, for example, Sticht 
(1982), Mikulecky & Strange (1986), Hargroves (1989), Haigler 
(1990) and Philippi (1988, 1991). 

The above examples are atypical. Mikulecky and d'Adamo- 
Weinstein (1991) observe that the majority of workplace literacy 
programs described in the available research literature tend to report 
no rigorous evaluation data* Many programs which do report 
evaluation data simply provide superficial information limited to 
surveys of learner satisfaction and anecdotal reports of effectiveness* 
Occasionally a pre and post administration of a standardized reading 
test (usually the T^st of Adult Basic Education - TABE or the Adult 
Basic Learning Examination - ABLE) provides an indication of learner 
gain in general reading ability- Only a few evaluations provide follow- 
up data on the impact of programs on learner job performance, 
retention, or earning power. 

Kutner, Sherman. Webb & Fisher (1991) recently reviewed 
workplace literacy programs funded by the U. S. Department of 
Education to determine the elements of effective programs. The 
authors examined 29 of 37 projects funded by the National Workplace 
Literacy Program to determine which programs were effective and 
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merited further examination in order to identify components of 
effective programs. The authors reported that: 

"Due to the absence of quantitative data necessary to identify 
particularly effective projects (i.e. improved productivity, low 
participant attrition, or improved test scores), study sites were 
recommended to OVAE staff. These sites were reported by 
project directors to have a high retention rate." (1991:26) 

Even in federally funded workplace literacy programs, for which 
program evaluation was an expectation for receiving funding, it was 
not possible to find six programs which had been rigorously evaluated 
for effectiveness. Selection of "effective" programs was based upon 
undocumented reports of retention from program directors. 

Formative and Summattve Evaluations 

It is possible to evaluate workplace literacy programs effectively 
using a combination of formative and summative evaluation. Formative 
evaluation of a workplace literacy program takes place at the beginning 
and during program operation. The purpose is to identify problem 
areas which can be addressed and modified while change is still 
possible and productive. Summative evaluation of workplace literacy 
programs takes place at the end of program operation and is designed 
to assess how well the program has succeeded. Assessing this sort of 
summative program impact requires gathering pre and post program 
data and then analyzing that data. 

Mikulecky, Philippi and Kloosterman have performed several 
such formative/summative evaluations using a version of Stufflebeam's 
(1974) Context, Input, Process, Product evaluation model modified for 
use with workplace literacy programs. In brief, the evaluation model 
employs the use of interviews, document analysis, observations, and 
test data to determine: 

1 ) the degree to which all involved with the program 
understand and share program goals; 

2) whether the resources in terms of personnel, 
materials, learning environment, and learner time 



17 2-J 



EVALUATING THE IMPACT OF WORKPLACE LITERACY PROGRAMS 

are sufficient, given current knowledge, to achieve 
the goals; 

3} whether the learning processes and methods 
employed are sufficient to accomplish the goals. 

These three evaluation goals provide information about the program in 
its formative stages. Results can be reported to program providers 
while there is still time to make program adjustments. A fourth, 
summative evaluation goal of this technique addresses: 

4) what evidence exists that program goals have 
actually been accomplished. 

Formative Evaluation 

A significant portion of the formative evaluation occurs early 
during program planning and operation (i.e. during formative stages of 
program development). Formative analyses usually employ interviews, 
the examination of program documents, and on-site observations to 
focus upon the degree to which program goals are shared, the 
adequacy of resources for achieving those goals, and the degree to 
which program execution appears to match stated program goals, 
i 

Program Goals 

Interviews, analysis of memos and planning documents, and 
early program observations often reveal that significant differences 
about program goals exist among funders, supervisors, instructors, 
materials designers and learners. Evaluation feedback during early 
program stages often initiates necessary clarification among program 
planners and participants. In some cases goals are expanded, in some 
cases goals are refined, and in some cases new vendors are sought. 

Examples of interview questions designed to reveal the various 
views of program goals among the participants and leaders follow 
below. 
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I Shared Goals 

1. What do you consider to be the main purposes, goals, and 
objectives of the basic skills training program(s)? 

2. Given the situation you find yourself in, what do you think are 
the most important things for an instructor to be doing? 



Additional information can be gathered from published program 
descriptions and from program planning documents. 

Resources 

Resources refers to the expertise of key personnel, the available 
instructional space and materials, as well as the time available for 
instruction. Early examination of resources sometimes reveals that 
resources are insufficient to accomplish goals espoused by program 
planners. Typical deficiencies are: 1) insufficient learner time to 
accomplish purported goals, 2) lack of appropriate learning materials 
or lack of resources to develop custom-designed materials which 
match workplace literacy program goals, and 3) difficulty in finding 
instructors with knowledge or expertise about workplace literacy 
requirements. Information about resources can be gathered by 
examining program facilities and from interviewing key program 
personnel. 

Examples of interview questions designed to elicit information 
about program resources follow below. 



Resources 

1. List training and experience you've had related to this job. 

2. What is your assessment of the following: 

Materials 
Facilities 

3. Please describe the following parts of the training program: 

Materials for diagnosing and testing learner abilities. 

How a learner's class and out of class learning time should be divided. 

How records are kept and what use is made of records. 
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Learning Processes and Methods 

We know from research reported in the previous chapter that 
literacy improvement takes a significant amount of time and that 
general literacy instruction is not very effective for workplace 
applications. Observation of classroom instruction, materials, and 
schedules sometimes reveals potential problems with the learning 
processes and methods offered by the program. Examples are: 1) 
Insufficient learner practice time with literacy or too much class time 
allocated to discussion, 2) Teaching general reading instruction with 
school books, off the shelf materials, or sometimes materials and 
activities selected because the instructor has found them useful in 
other settings, and 3) Little feedback from instructors about learner 
accomplishments (sometimes instructors do not or cannot comment 
upon what individual learners can and cannot do). 

Effective programs typically instruct learners using workplace 
related instructional activities and real or modified workplace 
materials. Teachers are familiar with job-literacy demands through 
direct observation or documented analyses of the jobs. When 
instruction using more general approaches or materials occurs, the 
teacher is usually able to relate the instruction to workplace needs. If 
instruction is not related to the workplace, it is because the program 
has simply elected to use a workplace classroom to address general 
literacy goals. In effective programs, no matter what the goal, 
sufficient learner practice time is available to allow reasonable 
expectation of success. Some effective programs even manage to 
expand practice time through homework. 

Examining the processes used in a workplace literacy program 
can be accomplished through classroom observation, examination of 
learner records and assignments, and through interviews with 
learners and instructors. A great deal can be learned by asking 
instructors and learners to describe how they spent time during the 
previous class period. Information from such interviews can help 
determine if learning activities and time allocation match program 
goals or if learning time is insufficient to meet these goals. 
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Classroom observation also allows the observer to gather 
information on how much time both instructor and learner spend in 
various activities. This information can then be analyzed to determine 
whether instructors are allocating time in ways that reinforce stated 
goals and in ways which are likely to be productive for learners, A 
form for recording such observational information follows below. 

Classroom Observation 

Time Student Activity Teacher Activity Comments 

0 
05 




Make note of time spent by students actually reading or doing things. Also note 
time learners spend listening to the instructor. When learners are in small 
groups or working individually should be mentioned. Special note should be 
made when the instructor or a student demonstrates how to do something. 



Summative Evaluation 

While the formative evaluation provides early information about 
the effectiveness of program operation, the summative evaluation 
provides information about whether the program achieved its goals. 

Evidence of Goal Attainment 

Well-evaluated workplace literacy programs gather baseline data 
before instruction begins. Typically data is gathered on the reading 
abilities, practices, and beliefs of learners. In addition, pre-program 
data is gathered on worker productivity or any other goal espoused by 
the program. Data-gathering is accomplished using formal tests, 
informally constructed tests related to workplace expectations, 
questionnaires, and interviews with learners and sometimes with 
supervisors. In addition, company records on productivity, safety, 
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attendance, and enrollment in subsequent classes can expand the 
evidence available for assessing program impact. 

This information establishes a base for later comparisons to end- 
of-program performance. At the end of the program, all learners are 
once again assessed using the same instruments. In some cases, it is 
possible to compare the performances of learners in a workplace 
literacy program to those of a control group of comparable employees 
who haven't yet been able to receive workplace literacy training. To 
do this, the control group takes pre and post assessments which 
parallel the instructional group. 

Program goals determine the types of information gathered to 
assess program impact. For example, if the program is to improve the 
ability of learners to perform more effectively in quality assurance 
groups, evidence needs to be gathered on such performance before 
and after training. If training is supposed to have a positive impact on 
learner reading habits at home and at work, these, too, need to be 
assessed before and after the program. 

Chapters 3, 4 and 5 will provide samples of methods and 
instruments for assessing the impact of workplace literacy programs 
on learner literacy abilities, practices, plans, and beliefs. In addition, 
methods for assessing the impact of workplace literacy programs upon 
productivity and upon the families of learners will be discussed and 
sample measures will be provided. 

Conclusion 

Only a few workplace literacy programs have been well 
evaluated, even though millions of dollars have been invested in their 
development and operation. To evaluate workplace literacy programs 
effectively it is desirable to perform both formative and summative 
evaluations. Formative evaluation takes place during beginning and 
middle stages of program operation and is designed to identify 
problem areas which can be addressed and modified while change is 
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still possible and productive. This type of evaluation usually involves 
the use of interviews, document analysis, and observations. Summative 
evaluation of workplace literacy programs usually takes place at the 
end of program operation and is designed to assess how well the 
program has succeeded. This sort of evaluation requires gathering pre 
and post program data and then analyzing that data. This implies 
using and developing measures directly related to program goals. 
Typical goals for workplace literacy programs include improved 
learner literacy abilities, improved literacy practices at work and 
elsewhere, changed learner beliefs about literacy, self, and education, 
and improved learner productivity on the job. 



Chapter 3 

Assessing Workplace 
Literacy Program Results 



Overview 

The summative evaluation of the Impact of workplace literacy 
programs is best performed using a combination of standard 
assessment tools and custom-designed measures. The custom- 
designed measures usually reflect the types of reading done on the job 
and in training courses. In addition, they can focus upon special 
objectives central to the workplace literacy program (e.g., increased 
productivity, comprehending safety information). This chapter will 
discuss several standard and custom-designed measures and provide 
examples. 

Among the topics discussed are: 

• the advantages and disadvantages of standardized tests; 

• the use of custom-designed measures, employing literacy 
task analyses to develop Cloze tests using reading passages 
from the workplace and job scenarios involving 
applications of literacy; 

• obtaining broader indications of adult literacy growth by 
monitoring changes in learners 1 literacy beliefs, practices, 
processes and plans. 

Model custom-designed measures, including Cloze tests, interviews 
and questionnaires are available in Appendices A - C, 

Standardized Tests 

Standardized reading tests are sometimes used in workplace 
literacy programs as a means of identifying the general reading 
abilities of learners. These tests often employ multiple-choice 
questions and short reading passages (from a few sentences to a 
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paragraph or two). Some are based on tests developed for use in 
elementary and secondary schools. 

The most commonly used tests are the Test of Adult Basic 
Education (TABS) and the Adult Basic Learning Examination (ABLE). 
Occasionally a workplace literacy program operating in conjunction 
with a Community College may use higher level general reading and 
study skills tests provided by the Community College. 

Advantages 

The advantages of using such tests are two-fold. Initially, the 
tests can provide information on the general reading abilities of 
potential learners. Many Community Colleges offering technical 
training courses, for example, will not enroll students with general 
reading or computational abilities below the eighth grade level. The 
National Assessment of Educational Progress results (Kirsch & 
Jungeblutt, 1986) indicated that approximately 20% of American 
adults read below the eighth grade level — including a significant 
number of adults who have graduated from high school. In some 
industries, more than half of the hourly employees score below an 
eighth grade level. Such individuals are prime candidates for basic 
skills support before and during technical training. Sometimes, 
standardized tests can be used to help identify such individuals. 

Secondly, standardized tests can be used as program pre and 
post assessments to measure gains in general reading abilities. 
Comparison of pre- and post-test scores can indicate the degree of 
effectiveness of a program. Also, post-test scores can indicate 
whether learners are ready to go on to textbooks and other general 
materials in technical training classes. These scores are generally 
indicative of how well someone can understand material with which 
he or she has little familiarity. For example, adults scoring at the 10th 
grade level on a standardized test would be very likely to have some 
difficulty with a textbook on an unfamiliar topic which was written 
above the 10th grade level. With some background knowledge on the 
topic, such people might be able to comprehend material a few grades 
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above their standardized test scores. It is extremely rare to find an 
individual who can comprehend material more than a few grade levels 
above his or her standardized test scores (i.e., even extensive 
background knowledge will nearly always be insufficient to allow a 6th 
grade reader to comprehend a manual written at the 11th- 12th grade 
level). 

Disadvantages 

The disadvantages of using only standardized tests in workplace 
literacy programs has been mentioned in Chapter 1. These tests 
measure general reading abilities and not the special sorts of literacy 
skills required in the workplace. A learner in a general basic skills 
class may improve in general reading abilities. For example, a learner 
could move from a 5th grade level (i.e. understanding the comics and 
very simple stories) to an 8th grade level (i.e. understanding the 
sports page and USA Today news stories). Though the improved 
reading ability may be of some use, the learner is not likely to be able 
to transfer those skills easily to reading an SPC chart, a technical 
manual, specialized work-orders, and industry-specific textbooks. The 
most efficient way to ensure improvement in these areas is to teach 
using these materials. Unfortunately, gains made in reading job- 
related materials may be only partially reflected in a standardized test 
which evaluates general reading skills. 

More subtle criticisms have been leveled against standardized 
test use for evaluation of workplace literacy programs. First, the 
quality of information revealed by these tests presents incomplete 
pictures of adult learning (as described above, an adult may read 
familiar materials somewhat better than general standardized test 
scores indicate). Second, the effects of such test scores on teaching 
are considered to be adverse by some (i.e., when teachers teach to the 
test and sometimes ignore materials that learners need for the job). 
Third, the way in which standardized test scores are reported can be 
humiliating to adults and counterproductive to learning. Some 
educators argue that when adults are informed that their 
performances are equivalent to low grade levels (i.e. 6th grade or 
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lower), it becomes a reminder of their failure rather than an objective 
description of abilities. 

Recommendation 

Some workplace literacy programs find that the disadvantages of 
standardized tests outweigh the advantages and rely instead on 
interviews, questionnaires, and other indicators to assess program 
effectiveness- Such assessments are custom-designed for the program 
being evaluated. Other programs use standardized tests as part of a 
mix of assessments. If standardized tests are used, they should never 
be the sole measure of learner gain in a workplace literacy program. 

Custom - Designed Assessments 

An alternative or a supplement to standardized tests are custom- 
designed instruments which are based on workplace materials and 
activities. To design such instruments and, indeed, to custom-design 
training programs, one first needs to determine how workers use 
literacy in a particular workplace. The first step is to perform a 
literacy task analysis. 

literacy Task Analysis 

Literacy task analysis is a way of identifying those aspects of job 
tasks which require reading and problem solving. These analyses are 
performed using a combination of observations of workers, interviews 
with top performers, and gathering samples of print used in the 
workplace and training classes. The goal is to determine the mental 
processes used by top performers as they solve problems and 
complete tasks which involve literacy. This information can be used to 
construct both test scenarios and instructional materials. It is 
important that these two be developed together, so that tests can 
assess what learners are really taught and both can be linked directly 
to the workplace. 

Observations and interviews with supervisors and workers are 
used to identify the areas in which performance needs to be improved. 
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Prime targets for literacy task analyses are tasks where basic skills 
deficiencies cost money or threaten health and safety- Other tasks can 
be identified by noting changes in the workplace (e.g., new 
technology, changed jobs or promotions) which confront some 
workers with new and sometimes troublesome literacy tasks. 

A good deal has been written about how to perform literacy task 
analyses (see Mikulecky, 1985; U.S. Departments of Education and 
Labor, 1988; Drew & Mikulecky, 1988; Philippi, 1988 & 1991). Most 
techniques involve determining the elements of a task and the 
strategies (both visible and mental) employed to accomplish the task. 
For example, filling in forms in some quality assurance procedures 
involves the elements of reading two-column charts, computing using 
decimals, knowing special vocabulary and abbreviations, and being able 
to summarize sequences of events. Within each of these elements, 
top performers employ a variety of strategies (i.e. skimming, 
estimating, interpolating, etc.) 



Philippi (1988) has identified 
strategies which are listed below. 

Vocabulary 

Recognize common words and 
meanings 

Recognize task related words with 
technical meanings 

Identify word meanings from sentence 
context 

Recognize meanings of common 
abbreviations and acronyms 



BfiCflgnlZlng ^use and ftffrrt predicting 
mitcomes 

Use common knowledge to avoid hazard 
or injury 

Apply preventive measures prior to task 
to minimize security or safety problems 

Select appropriate course of action in 
an emergency 



a number of such elements and 



Inferential C omprehension 
Determine figurative, idiomatic, and 
technical meanings of terms, using 
context clues or reference sources 

Make an Inference from text that does 
not explicitly provide required 
information 

Organize information from multiple 
sources into a sequenced series of events 

Interpret codes and symbols 



Literal comprehension 

Identify factual details or specifications 
within text 

Follow detailed, sequential directions 
to complete a task 

Determine the essential message of a 
paragraph or selection 
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Locating in formation within a text 



Using charts, diagrams and schematics 



Use table of contents, index, appendices, 
glossary, subsystems to locate 
information 

Locate page, title, paragraph, figure, or 
chart needed to answer questions or 
solve a problem 

Use skimming or scanning to determine 
whether or not text contains relevant 
information 

Cross reference within and across 
source materials to select information 
to perform routine activity 

Use a completed form to locate 
information needed to complete a task 
activity 



Comparing and contrasting 

Combine information from multiple 
sources 

Select parts of a text or visual materials 
to complete a task 

Identify similarities and differences in 
objects 

Determine presence of a defect or extent 
of damage 

Match objects by size, color, or 
significant marking 

Classify objects by size, color, or 
significant marking 

Distinguish between relevant and 
irrelevant information in texts or 
visuals 



Obtain a factor specification from a 
two-column chart to find information 

Obtain a factor specification from an 
intersection of row by column on a table 
or chart 

Use a complex table or chart requiring 
cross-referencing within text material 

Apply information from tables or 
graphs to locate malfunctions or to 
select a course of action 

Use simple linear path of an 
organizational chart to list events in 
a sequential order 

Use the linear path of a flow chart to 
provide visual and textual directions 
for a procedure, to arrive at a decision 
point or to provide alternative paths in 
problem solving 

Isolate each major section presented in 
a schematic diagram 

Isolate a problem component in a 
schematic and trace it to the cause of the 
problem 

Interpret symbols to indicate direction 
of flow, text points, components, and 
diagrammatic decision points 

Identify details, labels, numbers, and 
parts from an illustration or picture 

Identify parts from a key or legend 

Interpret drawing of cross-section for 
assembly or disassembly 

Interpret a three-dimensional, or 
exploded view, of an object for assembly 
or disassembly 

Follow sequenced illustrations or 
photographs as a guide 



Materials and information gathered during literacy task analyses 
can be used to develop instructional materials as well as to develop 
custom-designed assessment instruments for workplace literacy 
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programs. Examples of such instruments (i.e., job-related Cloze tests 
and literacy scenarios) are discussed below. 

Job-related Cloze Tests 

Standardized tests reveal an individual's general reading ability. 
A Cloze test is a custom-designed measure to assess how well a person 
can comprehend a particular type of reading material (e.g., job-related 
information). From the materials gathered during the task analysis, 
representative prose passages of about 150 words can be selected for 
the construction of Cloze tests. This is done by omitting every fifth 
word from a passage, usually leaving the first and last sentences intact. 
This results in a passage containing about 25 blank spaces which the 
test-taker is asked to fill in, using the surrounding context of sense 
and grammar. 

The ability of readers to replace missing words accurately 
correlates very highly with scores on traditional reading 
comprehension tests (Bormuth, 1969; Mikulecky & Diehl, 1980). A 
general rule of thumb is that being able to replace less than 35% of the 
missing words indicates that the passage is beyond the 
comprehension of the test-taker. In other words, if the reader can 
replace less than 9 out of 25 missing words, the reading is too 
difficult. Replacing 50% or more of the missing words indicates the 
ability to read and comprehend the material independently. So, in a 
passage with 25 blanks, a score of 13 shows that the reading is of a 
suitable standard for the reader. Scores between these values reflect 
the degree to which the reader needs some instructional help to 
comprehend fully what is being read. Nobody is ever expected to 
replace all the missing words correctly. A score of 50% is considered 
quite good, and making test-takers aware of this may defuse the 
frustration that they are likely to feel when unable to guess satisfactory 
words for a number of the blank spaces. 

A sample Cloze test (with answers included) is provided below. 
The instructions include a practice example, since many readers have 
never taken a Cloze test before and sometimes require guidance in 
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getting started. Appendix C includes instructions for developing Cloze 
tests and some additional examples * 



Name 



Date 



Cloze Exercise 



In a cloze exercise, you try to guess which words are missing* For 
example, in the sentence below, a word is missing* 



She looked before she 



the street* 



A good guess for the missing word is "crossed*" 
She looked before she crossed the street. 

In the story below, try to guess and replace the missing words. Don't 
expect to get them all. Some are nearly impossible* 

G.M Designs Safety for All Ages 
We all like to think about the old days* life seemed simpler and, in 

some ways, better then* But when it comes to , the 

good old days offer the same degree safety 



as today's cars 
technology 



_ trucks* Advancements in 
the G*M* vehicle you 



today among the safest 



the world* Each G*M* 



{continues to approximately 25 blanks) 



Cloze Exercise key: automobiUs, didn't, of, and, ma£t, purchase, in 

Job Problem Solving Simulations and Scenarios 

Simulations and scenarios can be constructed by using actual 
materials from the workplace to assess the job literacy abilities of 
workers. Information and materials gathered during the task analysis 
form the basis for constructing job-like scenarios in which the learner 
reads and makes decisions based on written materials. Scenarios are 
usually constructed to reflect a range of material types (i.e. prose, 
documents, graphic material), and sometimes involve both reading 
and computation. If the range of learner reading abilities is likely to 
be wide, it is useful to construct scenario questions which range from 
fairly easy to fairly complex, so that all test-takers can experience 
success at some level. 
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Appendix A contains samples of job scenarios and directions for 
constructing such scenarios. For full range testing purposes, it is 
recommended that scenarios include process questions, factual 
questions, inference questions, and application questions. Process 
questions determine how the reader reads a passage: that is, the range 
and sophistication of reading strategies employed. Factual questions 
should have answers based directly on the reading material, answers 
to inference questions can involve deductions from several places in 
the reading, and application questions should relate the reading to the 
interviewee's background knowledge. (See the examples below.) 

Process question 

I am going to show you a newspaper article about your industry. 

Explain to me how you would read this story in order to find out 
what the writer thinks. 

Describe what you would look at. What would you be thinking 
about? How would you go about reading this story? What would 
you do first, then next, then next? 

Factual question 

How many employees does ASMO have in Statesville? 

(Answer: 400. Listed in article ) 

I Inference question 
From the information provided about products, what do all four 
companies have in common? 
(Answer: All of them make some sort of motor. Requires the 
interviewee to search for commonalities not readily apparent. ) 
Application question 

What company makes products closest to your job at this 
facility? Why do you say so? 

(Answer: Relate a product on the list to what the employee 
makes. Requires the employee to sort through the 
information and then to apply it to his/her background 
knowledge. ) 
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In addition to their use in a pre-test to establish base-line data 
for assessment, job scenarios can be used at the beginning of a 
program to diagnose areas of learner difficulty. If the information on 
the scenarios is also part of a training curriculum, the scenarios can 
provide instructors with valuable information. For example, if a 
learner consistently has difficulty with inference questions across 
scenarios, the instructor can adjust instruction to provide more 
guidance and practice in this area. The instructor should not, 
however, provide detailed feedback to learners about their 
performance on the scenarios if the program intends to use those 
scenarios again as a post-test to assess learner gain and program 
effectiveness. 

A test can be used a second time to indicate learner growth if 
the learner has not been taught or given feedback using the actual test. 
It is also important that sufficient time has passed between pre- and 
post-tests (six weeks is usually sufficient) for detailed memory to 
decay. If such time is not available, it is possible to develop two very 
similar tests and establish the comparability of the two scenarios by 
noting how a pilot group scores on them. This is a fairly lengthy 
procedure, but worthwhile if the tests will be used with many learners 
for several years. Once comparability has been established, the two 
forms of the scenario can be used as pre- and post-measures. 
However, using the same scenarios for both tests provides a more 
reliable means of establishing comparability. 

Assessing a Broader Conception of Adult Literacy Learning 

Lytle (1990 a & b) suggested that performance measures (tests 
and exercises) miss a good deal of important information about adult 
literacy learning. In addition to gains in literacy skills, adults may 
make changes in what they believe, how they behave, and in their 
aspirations. Lytle suggests several dimensions which constitute a 
fuller understanding of adult literacy and adult literacy growth. These 
dimensions are learner beliefs about literacy and themselves, learner 
literacy practices, the literacy processes employed by learners while 
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reading, and the plans a learner has which may involve literacy use. 

Lytle's conceptual framework has been adapted to the present 
workplace literacy project to test the importance of these aspects of 
adult learning: beliefs, practices, process and plans, and in order to 
seek out ways to enhance learning. Information about these 
dimensions of learner literacy were gathered using a combination of 
questionnaire items, interview questions, and requests that learners 
explain their literacy strategies or processes while doing job scenarios. 

Beliefs 

In the interview, learners were asked to describe themselves as 
readers and writers and to describe someone who seemed to be very 
good at reading and writing. They were also asked to provide reasons 
for their answers. Changes in these beliefs are likely to precede 
changes in literacy abilities. Sample questions from the interview 
follow below and are available in Appendix A. 



Beliefs 

1. Describe someone you know who is good at reading and writing. 
What makes you choose this person? 

2. How good do you consider yourself to be at reading and writing? 
What makes you think so? 

3. Describe how you would like to be in terms of reading and writing. 
{Probe : Could you give me some examples?) 



Practices 

Learners were asked orally and in the questionnaire for 
information about the types of reading and writing they do on the job 
and off the job . They were asked to rate the difficulty they had in 
reading each item on a list that included books, signs, training 
manuals, pay stubs, charts and cartoons. They were also questioned 
about the frequency of their literacy-related activities: how often, for 
example, they read a newspaper, made a shopping list or visited a 
library, as well as how many books they owned. Information was also 
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sought about literacy practices in workplace situations which ranged 
from departmental meetings to handling broken equipment, from 
reading instruction manuals to reading a health insurance policy. 

Sample interview and questionnaire items follow below and are 
available in Appendices A and B. 



Practices 

Interview item 

Tell me the sorts of things you read and write away from work 
during a normal week. 

(For probe, ask: "Can you give me more examples?") 
Questionnaire items 

1. First check only the things you've read in the past month . 

Now go back and rate your ability to read the items you've checked. 

poor excellent 



local newsDaDers 
classified ads 


1 z 
1 2 


o 

3 


4 5 


telephone bills 


1 2 


3 


4 5 


TV guide listings 


1 2 


3 


4 5 


magazines 


1 2 


3 


4 5 



2. In the last 7 days how many times have you read a newpaper? 

0 123456789 10+ 

3. You talk a lot in team or department meetings, asking questions 
or sharing ideas. 

very like me 1 2 3 4 5 very unlike me 



Process 

In order to seek information about the processes which learners 
go through when reading work materials developed as part of the job 
scenarios (see description on pp. 32-34 above), some questions in 



36 



ASSESSING WORKPLACE LITERACY PROGRAM RESULTS 



each scenario asked students to think aloud about the way they were 
reading the material. The purpose of these questions was to 
determine whether learners were employing sophisticated reading 
strategies (i.e. skimming, focussing, asking questions, etc.) and 
whether the choice and use of reading strategies improved as a result 
of training. Sample questions follow below and are available in 
Appendix A. 



Process 

I am going to show you a newspaper article about your industry. 

Explain to me how you would read this story in order to find out 
what the writer thinks. 

Describe what you would look at. What would you be thinking 
about? How would you go about reading this story? What would 
you do first, then next, then next? 

I 

Plans 

Some questions in the interview sought information about the 
learners' plans, especially in relation to education and goals requiring 
increased literacy abilities. These questions asked for information 
about learner plans for 1 year, 5 years, and 10 years ahead. Sample 
questions follow below and are available in Appendix A. 



Plans 

Now I'd like to ask you about your plans. Explain how you see 
reading and edu cation as part of these plans: 

A What are your plans for the next year? 
B. What are your plans for the next 5 years? 

C What are your plans for the next 10 years? 

4T 
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Conclusion 

Workplace literacy program impact is best measured using a 
mixture of standard assessment tests and custom-designed 
instruments. Standardized tests provide useful information about 
general reading ability, but may be misleading with regard to 
workplace literacy skills. 

Custom designing starts with a literacy task analysis to identify 
aspects of job tasks which require reading and problem-solving, and in 
which performance needs to improve. Cloze tests based on workplace 
materials can be used to assess workers' abilities at job-related 
reading. Job scenarios can test their skills in using what they read, 
through process, factual, inference and application questions. 

A broader conception of adult literacy learning can be assessed 
by seeking information about the learners' literacy beliefs, practices, 
processes and plans, using interviews and questionnaires. 
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Chapter 4 

Assessing Impact on Family Literacy 



Overview 

Chapters 2 and 3 have considered the evaluation of workplace 
literacy programs in relationship to impact at the workplace. 
Workplace literacy programs also have effects on workers' families and 
children. This chapter examines means for evaluating those effects. 

The chapter considers the factors which can be used to measure 
family literacy impact. A review of previous research on this topic is 
followed by a discussion of questionnaire and interview items used in 
the current evaluation of family impact of workplace literacy programs. 
The complete instmments appear in Appendices B and D. 

Topics discussed are: 

• socio-economic level of parents; 

• education level of parents; 

• aspiration of parents for their child's education; 

• ability of parents to act as role models; 

• promotion by parents of literacy activities. 

Workplace literacy programs and family literacy 

It is possible for workplace literacy programs to affect not only 
learners' literacy levels and productivity on the job but also literacy in 
the workers' families. Home literacy activities can both benefit the 
employees' children and also increase the employees' literacy practice 
time. Program descriptions provide many anecdotal examples of these 
benefits. A young mother in a workplace literacy program at Planters 
LifeSavers in Virginia reports that she enrolled in the company's basic 
education program, not only to be able to help her seven children with 
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their homework, but also to persuade her oldest son that it is 
important to finish school (Cooper et al., 1988). Gross. Lee and Zuss 
(1988) reported that one workplace literacy student began to help her 
eight-year-old son with homework and was able to leave handwritten 
messages for her children. 

The effect of literacy programs on the children and families of 
workers is often neglected in evaluating program effectiveness, 
however. At both sites in the current study, assessment of family 
literacy was conducted through pre- and post- questionnaires 
modified from survey questions used by Greer and Mason (1988). The 
questions covered parental guidance, literacy artifacts, and child- 
initiated literacy behavior. In addition to individual questionnaires for 
parents, some parents were interviewed in focus groups. The Family 
Literacy Focus Group Interview was administered only to the 
participants of the program at one site and was based on the work of 
Fitzgerald, Spiegel, and Cunningham (1991). Samples of the questions 
from each of these instruments will accompany the discussion of the 
impact of parent literacy on their children which follows. 

Impact measures 

At least five factors have been identified by research as related to 
the ability of parents .to affect a child's achievement in literacy: the 
socio-economic status of the parents, their educational level, the 
aspirations parents have for their child's education, the ability of a 
parent to act as a role model, and the parents' promotion of literacy 
activities. Some of these factors are more easily altered than others 
through a workplace literacy program. 

The first two, the direct correlation between the educational or 
socio-economic levels of parents and the child's literacy ability, have 
been identified by researchers as solidly linked (Chall, 1984; Laosa, 
1984; Sticht, 1983; Sticht & McDonald, 1990). However, a brief 
workplace literacy program is not likely to affect income and general 
education levels directly nor very quickly. The latter three measures 
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are more likely to be affected by a workplace program. 

Parents' aspirations for the best education for their children 
appears to be important in the child's own aspirations, as 
Marjoribanks found (1984a & b). Chall and Snow (1982) have shown 
that children whose mothers set high educational goals for them 
achieve higher levels of reading comprehension and word recognition. 

Some research indicates that high educational aspirations for 
one s children may be connected to a parent's own educational level. 
Laosa (1982) found a significant relationship between a mother's 
educational aspirations for her child and the level of schooling of both 
parents. However, Lujan & Stolworthy (1986) found that the 
educational aspirations of lower socio-economic status families were 
just as sincere and ambitious as those of parents from middle to 
higher levels. Unfortunately, as important as having aspirations may 
be. parents who are unable to help their children reach such goals are 
at a disadvantage. 

The ability to model reading and to engage in interactions with a 
child which encourage and teach literacy is important. However high 
the aspirations of a parent might be, Nickse et al.(1988) suggest that 
illiterate adults cannot model what they do not know. Fitzgerald et al. 
(1991) found that low-literacy parents did not even mention adult role 
modeling as important in interviews about how to help their children, 
whereas high-literacy parents talked about the need to have their 
children see them reading. Work with middle school students by 
Fielding, Wilson, and Anderson (1986) showed that readers among 
students tended to have parents and siblings who read. A parent's 
ability to model oral language skills also seems to affect a child's ability 
to read in school (Sticht, 1983; Loban, 1964; Chall & Snow, 1982). 

In the current study, questionnaire and interview items were 
developed to measure effects in these areas. Examples follow below. 
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Questionnaire item 

In the last 7 days how many times has your child seen you 
reading or writing? 

0 1 2 ~3~ 4 "5" 6 7 8 9 10+ 
Interview item 

At home, do your children see you doing any reading or writing? 
(i.e. books, magazines, papers, recipes, directions, letters, lists, 
notes, etc.) 



Closely related to the parent as a role model is the activity of a 
parent to encourage a child in literacy activities. Included in such 
activities are the creation of a literacy environment in the home and 
the use of a community library. According to Fielding, Wilson and 
Anderson (1986) readers in middle schools come from homes in 
which there are many books and many opportunities to go to a library. 
Similarly, Greer and Mason (1988) found that the children who score 
higher on tests of reading recall are children who frequent a library, 
have someone at home who reads to them often and helps them read, 
and who have books and magazines purchased for them. 

Parents who directly promote their children's reading have 
children who seem to do better in school. McCormick and Mason 
(1986) sent simple-to-read Little books home for parents to read to 
their preschool children. The parents were given instructions in 
helping their child learn to recite. That activity had a significant 
effect on the child's later reading in kindergarten and first grade. 
Furthermore, Chall and Snow (1982) discovered that reading 
comprehension was higher for the second, fourth, and sixth graders 
they studied if their homes provided more literacy experiences and 
reading materials which were both interesting and appropriate for the 
child. Stewart (1986) administered a reading test to 56 children and 
compared their scores to the answers their parents had given to a 
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questionnaire that assessed home support for early reading. He found 
a significant relationship between borrowing books from a public 
library and the children's performance on the test. 

Assessing the literacy environment in terms of reading materials 
available in the home or trips to the library appears in questionnaire 
items. 



1. In the last month how many times have you bought or borrowed 
books for your child? 

0 1 2 ~3 4 ~5~ 6 7 8 9 10+ 

2. In the last month how many times has your child gone to a 
public library? 

0 1 2 ~3 4 5~ 6 ~7 8 9 10+ 
1 

Being a role model and providing materials are not the only ways 
parents improve children's literacy. Activities in which parents and 
children interact together are also important. Such activities include 
reading aloud to a child, encouraging the child to ask questions and 
make predictions about the text, allowing the child to initiate a 
literacy event, and parental involvement with the school. Both 
questionnaire items and questions from the Family Literacy Focus 
Group Interview address such activities. 

Time spent reading with a child, particularly prior to" the school- 
age years, can affect the child's later success or failure in reading. 
Stewart (1986) visited the homes of four children several times over a 
two-month period and learned that stimulation from parents made 
more of an impact on children's reading abilities than merely having 
several books around the house. In fact, the effect of reading aloud to 
children has been widely studied. Chomsky (1972) revealed that the 
most important activity for building the knowledge required for 
literacy success is reading aloud to children. Laosa (1982) found 
significant correlations between mothers who read to their children 
and the child's literacy skills in preschool. Buchanan-Berrigan (1989), 
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Anderson (1985), Teale (1984), Teale & Sulzby (1986), and 
Fitzgerald, Spiegel and Cunningham (1991) also indicate that reading 
aloud to children helps the development of preschool literacy, which, 
in turn, enhances school learning, especially when the child is an 
active participant. 

Below are questionnaire items which assess such activity: 



1. In the last 7 days how many times have you read/looked at books 
with your child or listened to him/her read? 

0 1 2 ~3~ 4 ~5~ 6 7 8 9 10+ 

2. In the last 7 days how many times have you helped your child 
with homework and/or with school projects? 

0 1 2 ~3~ 4~~~~5~ 6 7 8 9 10+ 



Several studies have revealed that parents who have been 
observed reading to their young children are also found to encourage 
them to label pictures, ask questions, and relate text information to 
their own experiences (DeLoache & DeMendoza, 1985; Harkness & 
Miller, 1982; Snow & Ninio, 1986; Pellegrini, Brody, & Seigel, 1985; 
Yaden, 1982). As Mason & Stewart (1988) suggests, these parents are 
leading their children towards the use of inference and 
comprehension monitoring strategies. The benefits of reading aloud 
to children, therefore, seem to be greatest when the child is an active 
participant who engages in discussions about stories, learns to identify 
letters and words, and talks about the meaning of words (Anderson, 
1985). 

An interview question assessing such interactions follows below: 



Do you do any reading or writing activities with your children? 
(i.e., visit library, hear stories, read to them, watch educational 
television, look at magazines or books with children, 
point out words to them, play school, show them how 
to read or write, etc) 
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From reading with a child and encouraging the child's 
interaction with the text, a next step is to attend to whether the child 
ever initiates the reading activity. McCormick and Mason (1986) 
found that those parents provided with inexpensive books for their 
children reported significantly more child-initiated use of books and 
child-initiated attempts to print than did a control group. More 
importantly, they indicated that the children had invited their parents 
into literacy activities, such as asking to read stories to their parents 
and asking for help with new stories, to a greater extent than the 
children of a second group of parents who were not given books. 
Teale (1983) discovered that as children become more adept, they 
take over more and more of the interaction until they can read the 
book alone or write on their own without help. 

Child-initiated behavior was more thoroughly examined by Lujan 
and Stolworthy (1986), who found that the most significant result 
from parent training was a positive change in most children's literacy 
behavior. For example, the children began to attend more closely to 
story time and parent instruction. They showed increased self- 
direction in organizing personal time so that there would be time at 
night for story reading. 

Questionnaire items addressing these issues follow below. 



1. In the last 7 days how many times has your child looked at or 
read books or magazines? 

0 f 2 ~3~ 4 ~5~ 6 ~7~ 8 9 10+ 

2. In the last 7 days how many times has your child asked to be 
read to? 

0 f 2 ~3~ 4 ~5~ 6 7 8 9 10+ 

3. In the last 7 days how many times has your child printed, made 
letters, or written? 

0 f 2 T 4 ~5~ 6 7" 8 9 10+ 
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CONCLUSION 

Workplace literacy providers want to get the most for their 
investment. Effective programs may be able to improve the abilities of 
workers on the job while at the same time providing a benefit to 
children in the home. Longer term effects of increasing workers' 
literacy abilities can include examining the effects on a worker's family 
and children. Yet workplace literacy program evaluations often 
neglect such impacts. We know, too, that as workers are encouraged 
to carry newly-won literacy abilities home, they benefit from the 
opportunity to increase their own practice of these skills. 

In assessing the effects of workplace programs on workers' 
families, five factors have been identified. These are: 

1) socio-economic status of the parents: 

2) parental educational level: 

3) parents' aspirations for their child's education; 

4) the ability of parents to model literacy practices: 

5) parental encouragement of literacy practices with their 
children. 

The first two are not as readily affected by short-term workplace 
programs and, therefore, are less desirable assessment targets. 

Measurement of parental aspirations, modeling, and 
encouragement were conducted during the current study through 
questionnaires given before and after the program and through Family 
Literacy Focus Group Interviews. 
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Chapter 5 

Assessing Impact on Productivity 



Overview 

A review of the literature on productivity assessment shows that 
little is known about the effect of workplace literacy programs on job 
performance, but there is some evidence of the value of such programs 
and of the costs associated with lack of training. Methods that can be 
used to assess workplace literacy programs include the following. 

A training program can be assessed for its impact on 
productivity using employee output, and such indicators as safety, 
absenteeism and retention, with these measures being taken both 
before and after training. Also, employees can be rated by their 
supervisors on various aspects of job competence and attitude, and 
changes in these ratings could be used in the calculation of the dollar 
value of the program to the company. 

Such methods and others directly related to literacy were 
incorporated into this study. To measure changes produced by a 
program, the following instruments were used both before and after 
training: 

■4 

• Employee performance was assessed using records of 
absenteeism, safety, discipline, grievances and suggestions. 

• Interviews and questionnaires were used to assess employee 
practices and processes of job-related literacy. 

• Supervisor ratings were obtained on various aspects of 
employee lob competence and attitude. 

Literature on productivity assessment 

Workplace literacy programs have been offered by many 
organizations, both government and private, but not much is known 
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about the effect of such programs on the job performance of the 
employees involved- For the most part, these organizations have 
regarded literacy programs more as philanthropic than as business 
enterprises, and so have not considered it appropriate to subject them 
to their usual cost-benefit analyses* 

There are, however, a number of indications that such programs 
can have a positive influence on the effectiveness of the workers 
involved* Collino et al (1988: 19, Note 17} mention a Blue Cross/Blue 
Shield program that has decreased turnover, improved performance 
and promotion prospects, as well as increasing motivation and self- 
confidence among employees. Also, the Federal Reserve Bank's Skills 
Development Center has had considerable success in training tinder- 
educated school dropouts up to a standard of job performance 
comparable to qualified entry-level workers (Hargroves, 1989). 

In addition, there are a number of cases on record of the costs 
associated with employees* lack of basic skills: 

• the inability to read a ruler wasted $700 worth of steel in one 
morning; 

• the inaccurate use of new scheduling equipment cost $1 m to 
correct the resulting errors; 

• employees at a lumber camp imitated the illustrations on safety 
posters because they could not read the text describing these 

as dangerous practices to be avoided (Collino et al, 1988: 11-12). 

However, the fact remains that, as noted earlier in this book, 
there has been very little systematic evaluation of workplace literacy, 
even of its effect on the employees' more general ability to cope with 
everyday literacy demands. So it is perhaps hardly surprising that two 
recent reports should come to the following conclusions. 

,f Very little research exists about the relationship of literacy to 
job performance. Much of what exists is sketchy and based on 
information obtained from studies conducted in the military" 
(U.S. Depts of Education and Labor, 1988: 37). 

Collino et al (1988) found that, even when companies do conduct 
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assessments of their literacy programs, the results are not made 
public. Furthermore, such assessments rarely involve a study of how 
productivity might be affected: 

"At best management relied on informal feedback of supervisors 
regarding employee performance" (Collino et al, 1988: 9). 

Methods of use for workplace literacy programs 

A workplace literacy program should have a positive and 
measurable impact on productivity. However, most companies do not 
have an evaluation methodology and therefore can not easily recognize 
the impact on productivity of training workers. 

Impact on productivity 

Though little research exists on methods to assess the impact 
on productivity of workplace literacy programs, more research and 
discussion is available on the general topic of the impact of training 
upon productivity (National Research Council, 1979). When workers 
are producing an actual physical output, any change in the quantity or 
quality of that output can be measured for the same workers before 
and after training or a comparison can be made between the output of 
trained and untrained workers. Programs that make such assessments 
are usually broad range training programs which can compare the 
output of a trained plant, division or work-team to a comparable 
control group. Assessing productivity impact at levels below the work- 
team is often precluded because many industries do not collect 
productivity information (i.e. production and defect rates) at the 
individual level. 

A broader definition of productivity allows for some information 
to be collected at the individual level. For example, other factors that 
may be affected by a training program are 

• retention and promotion; 

• absenteeism and punctuality; 

• dishonesty; 
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• accident rates; 

• use of suggestion boxes. 

In addition, if productivity is broadly defined as supporting 
corporate goals, increased participation in voluntary activities (e.g., 
additional training or employee quality participation groups) can also 
be included among productivity indices- (See for example Collino et 
al, 1988; U.S. Depts of Education and Labor, 1988.) AD of these 
factors can be made the subjects of comparison, both of employees 
before and after a program and of those employees with others not 
attending the program. 

Supervisor ratings 

Another way of obtaining information about the effect of training 
on individual workers is to use supervisor ratings. These can be a 
single score for each employee or, preferably, a set of scores covering 
various specific skills and attitudes associated with job performance. 
Depending on the nature of the work concerned, these aspects are 
likely to include: 

• setting up and operating machines; 

• keeping up-to-date with paperwork; 

• taking responsibility for their own work; 

• having the initiative to solve problems as they occur; 

• communicating with other workers; 

• being committed to company goals. 

For each aspect, a rating scale can be set up with descriptions of 
worker performance at low, average and high levels. For example, in 
order for supervisors to rate, on a scale from 1 to 10, workers 1 
initiative in dealing with machine errors, the descriptions might be: 



rating of 2 - ignores machine errors and lets them build up 
rating of 5 - realizes machine errors and attempts solution 
rating of 8 - monitors machine errors and deals with them 
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These descriptions serve to anchor the rating scale to specific worker 
behaviors, in order to produce consistent ratings between supervisors 
and from the same supervisor in pre- and post-training assessments. 
Developing the descriptions with the help of workers and supervisors 
enables them to be a realistic reflection of job practice. For examples 
of supervisor ratings, see Appendix G. 

Such job performance scales anchored to validated behaviors 
have proven to be useful in lowering error, increasing reliability, and 
being efficient in terms of job performance ratings (Borman, 1977; 
Latham, Wexley & Pursell, 1975). Job performance scales anchored to 
behaviors have proven to be most effective when special care is taken 
in describing the job dimensions to be evaluated (Dickinson, 1977) 
and when unambiguous anchor descriptions are developed with 
involvement from job incumbents and the supervisors who are to 
participate in rating job performance (Norton, Balloun & 
Konstantinovich, 1980). Mikulecky & Winchester (1983) and 
Mikulecky & Ehlinger (1986) have successfully used such anchored 
supervisor ratings to assess job performance in the nursing profession 
and the electronics industry. 

An alternative supervisor rating approach is to use an overall 
assessment of the performance of each employee, as rated by their 
supervisors, to calculate the "Utility" of the training or literacy 
program in terms of its benefits minus its costs. (See Sheppeck & 
Cohen, 1985; Schmidt, Hunter & Pearlman, 1982; Cascio, 1982.) For 
this calculation, other factors required are an estimate of the 
difference in dollar value to the company between an outstanding and 
an average employee, the likely duration of the training's effect, and 
the cost of the program. (See the Endnote to this chapter for an 
example of a utility calculation and further details concerning the use 
of this method.) 
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Methods incorporated into this study 

For this project, indicators relating to productivity were 
gathered on each employee both before and after the training program 
or, in the case of on-going programs (such as GED classes), at suitably 
spaced intervals* These included statistics on attendance, safety and 
suggestions. Interviews with employees and questionnaires that they 
filled out assessed their attitudes to the workplace and various job- 
related skills. Also, supervisors assisted in the development of 
anchored rating scales which they then used to assess each employee 
before and after training. 

Productivity data 

It was not possible to obtain data on the actual output of the 
individual employees involved in training, for two different reasons at 
the two companies participating in this project. As mentioned above, 
companies do not gather such data for units below that of the work 
team; so, at both sites, the attempt was made to have a whole team 
take part in training at the same time. However, at one site, the 
composition of the class could not in the end be arranged in that way, 
as teams were re-organized and some individuals could not be released 
for training. And, at the other, although a team did go through 
training together, data for that team could not be separated out from 
plant-wide figures for individual analysis. Thus, in order for the 
gathering of output data to be successful, it must be possible for a 
company to arrange training for a whole work team and for 
mechanisms to be put in place, perhaps specially for this purpose, to 
obtain the output data for that team. 

The following measures were, in fact, used to evaluate changes 
in employee performance, each measure being taken both early and 
late in each program for the employees involved, so as to assess the 
impact of that program. In addition, comparisons were made in one 
case with a control group of employees who had not yet participated in 
that program. 
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Data relating to employee attitudes were collected on: 

• absenteeism; 

• grievances submitted; 

• discipline records; 

• workplace safety records; 

• suggestions made; 

• suggestions accepted. 

Interviews and questionnaires 

To supplement these company records, employees were 
interviewed and also filled out questionnaires (see Appendices A and 
B). Their purpose was, in part, to assess attitudes toward the 
workplace and competencies associated with the workers' jobs. In the 
interviews, the employees were asked about the types and amounts of 
reading and writing they do on the job; this was to assess the quantity 
and quality of their workplace literacy activity. They were also asked 
to demonstrate specific skills related to their work, such as using a job 
aid or written information sheet, and a graph or table. Questions here 
were of two types: process and content. Process relates to how the 
workers go about using the item; for example, 

• do they use job aids regularly; 

• what parts of them do they look at; 

• how long does it take them to read one. 

Content questions are more specific to the particular item, asking for 
information that the workers should be able to obtain from the sheet 
in front of them, such as: 

• what components do you need to make this part; 

• how do you carry out this procedure; 

• what does this graph show as the inventory value on a certain 
date. 

Some content questions called for interpretation by the interviewee, 
drawing on the given information to make inferences about the 
situation; for example. 
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• why do you think the value fell during this particular month?; 

• what might have caused this type of wastage to occur? 

The questionnaire dealt, in part, with reading and talking in 
Relation to the workers 1 jobs, particularly their abilities and confidence 
in reading instructions and talking in meetings. Among items of a 
more general nature that they were asked to rate as easy or hard to 
read were a number of work-related ones: 

•job aids; 

• part specifications; 

• safety rules; 

• benefit information; 

• the plant newspaper. 

In addition, they were asked to use a scale from Very like me" to "very 
unlike me 11 in rating such statements as: 

• your ideas are often discussed in meetings; 

• when written information is handed out, you read it to 
see what it is about; 

• when paperwork comes to you about your job, you often have 
trouble reading it. 

Supervisor ratings 

To obtain another perspective on this information gathered 
directly from the employees, supervisors were asked to assess each 
worker on various aspects of job performance that contributed to 
productivity and were related to task competence, communication, 
teamwork and paperwork skills. These assessment instruments were 
developed with the assistance of those who would be using them, 
through interviews that determined what aspects should be covered 
and how to describe behaviors typical of top, average and bottom 
performers. Specific aspects included were the ability to 

• set up and calibrate a machine; 

• use recording forms; 

• trouble-shoot machine errors. 
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Also assessed were attitude indices such as: 

• how much they took responsibility for their own work; 

• how well they worked as a member of a team; 

• how committed they were to company goals. 

For each of these aspects, anchoring descriptors for Bottom, Average, 
and Top performance were related to a scale from 1 to 10. guiding the 
supervisors in making their assessments. 

Thus, the final supervisor rating form could contain instructions 
such as: 

• An average employee would be rated 5. 

• A top employee would be rated 8 or higher. 

• A bottom employee would be rated 2 or lower. 



And one item on a form could appear as follows (see Appendix G for 
further examples). 



Paperwork 

Bottom 

intimidated 
by job-related 
paperwork and 
does it poorly 



Average 

does job-related 
paperwork, 
simply 
keeping pace 



Toe 

completes all 

job-related 

paperwork 

and tries to 

improve 

procedures. 



i"""2 3 ~4~" 5" 6 7 8 9 10 
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CONCLUSION 

In order to assess the impact of a workplace literacy program on 
employer objectives, measures of productivity should be taken before 
and after training. Such measures include company records, employee 
interviews and questionnaires, and supervisor ratings. 

Company records can supply information on output, safety, 
dishonesty, discipline and grievances, absenteeism and punctuality, 
retention and promotion, and productivity suggestions. 

Employee interviews and questionnaires can supply information 
on attitudes and job practices and skills. These include how much 
reading and writing employees do in the workplace, how competent 
they are at various types of reading, and their confidence with reading 
and in meetings. 

Supervisor ratings can also supply information on employees' 
job-related skills and attitudes. Using anchoring descriptors for top. 
average and bottom performers, rating scales can be developed to 
cover such aspects as task competence, communication, team work 
and paperwork skills. 



Endnote - Calculation of "Utility" of Training. 
Calculation of the "Utility" or cost effectiveness of a training program requires: 

1. An overall measure of the job performance of each employee trained and of a 
comparable group of untrained workers. (This could be either a supervisor rating or 
be based on production outcomes.) 

2. A measure of the dollar value to the company of the difference between outstanding 
and average employees. (This estimate of the standard deviation of performance is 
known as the "value".) 

3. The expected duration of the training's effect. 

4. The cost of the training. 

As an example of a "Utility" calculation, let us suppose that the 20 employees 
who have completed a training program are rated by their supervisors, on average, at 6 
out of 100 The untrained employees received an average of 50. with a standard 
deviation of 10. The trained workers are at a level of 1.5 standardized units above the 
untrained -- this is the performance difference. If it is estimated that the average 
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employee Is worth $18,000 to the company and an outstanding one is worth $26,000, 
then an estimate of the 'Value" or standard deviation of employee performance is 
$8 000 (the difference between these two amounts). Suppose also that training costs 
$2 000 per employee and the effect of training is likely to last 3 years. Then we have: 



| "Utility" = 

Years duration x Number x Performance x Value - Number x Cost per 
of effect trained difference trained trainee. 

3 x 20 x 1.5 x $8,000. - 20 x $2,000 

= $720,000 - $40,000 
= $680,000 net utility to the company 



This formula was originally developed by Brogden (1949) and revised into its 
present form by Schmidt, Hunter, McKenzie & Muldrow (1979). For examples of its use, 
see Sheppeck & Cohen (1985), Schmidt, Mack & Hunter (1984), Schmidt, Hunter & 
Pearlman (1982) and Cascio (1982). 

Criticisms of the method are to be found in Boudreau (1983) and Cronshaw & 
Alexander (1985); these are refuted by Hunter, Schmidt & Coggin (1988). Modifications 
to the procedure are suggested by Bobko et al. (1983) and Cascio & Ramos (1986); see also 
Cascio (1982). Comparative studies of such modifications are contained in Greer & 
Cascio (1987), Burke & Frederick (1986), and Weekley et al. (1985). 



Part II 



THE CURRENT STUDY 



Chapters 
Structure of Current Study 



Overview 

The purposes of this study were to develop an impact 
assessment model for workplace literacy programs and to produce 
data on the impact of programs at two sites, A secondary goal was to 
refine the model for use at other sites. 

The two sites chosen are very different, but both operate 
established, effective and diverse programs, involving technical and 
communications training, GED and ESL classes. 

Pre- and post-program data were gathered on learner job 
productivity, learner literacy attributes, and literacy practices with 
their families. Instruments and methods used to gather data included: 

• learner interviews, tests and questionnaires, based on Lytle's 
"Beliefs, Practices, Process, Plans" adult literacy model; 

• questionnaire items based on key practices for developing home 
literacy; 

• productivity indicators such as attendance, safety and supervisor 
ratings of on-the-job use of literacy and communication skills. 

The data were analysed using statistical comparisons of 
quantitative information, as well as qualitative and quantitative analyses 
of categories emerging from open-ended responses to interview 
questions. 

Purpose 

Although federal and private support funds thousands of 
workplace literacy programs, very few programs have been evaluated 
beyond a superficial level (Mikulecky and D'Adamo-Weinstein, 1991). 
Typical workplace literacy program evaluations involve anecdotal 
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reports, learner satisfaction questionnaires or pre and post results 
from a standardized basic skills test such as the T*A,B,E, or the A*B,L,E, 

In late 1990 f the National Center on Adult Literacy funded a 
project to develop and pilot a model for evaluating the impact of 
workplace literacy programs. During Year 1, parallel pilot studies of 
two workplace literacy programs were used to: 

(1) develop an impact assessment model for workplace 
literacy programs; 

(2) produce data on the impact of two quite different 
workplace literacy programs. 

The goals of this first year's efforts were to refine the impact 
evaluation model so that it could be transferred to additional sites 
during subsequent years and to establish base-line data for the level of 
impact one could expect from established workplace literacy 
programs. 

Populations 

The evaluation model to assess the impact of workplace literacy 
programs was piloted at two sites: Delco Chassis of Rochester, New 
York and Cumberland Hardwoods of Sparta, Tennessee, Though the 
sites were chosen for their differences in size, demographics, location 
and industry, each site has a well-established workplace literacy 
program which addresses several different populations (i,e, technical 
communication and basic skills training, G,E.D, preparation, and 
English as Second Language preparation at Delco), Leaders at both 
companies see it as necessary for survival to increase employee 
involvement in the decision-making processes of day-to-day business. 
Each firm intends that those actually producing the goods will be able 
to decide, for example, whether machines require adjustment or 
whether their production line has stockpiled a sufficient quantity of 
product X and should switch to product Y, 

Both companies run education programs judged by regular state 
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and federal acknowledgement to be effective, model examples of 
workplace literacy education. Since we were piloting new 
instruments for assessing the achievements of such programs, we 
wished to establish benchmarks with programs which had been 
independently judged to be good programs. 

Classes and individuals at each site provided information through 
interviews, tests, checklists and questionnaires to assess the impact of 
programs upon learners, their productivity, and family literacy in 
learners' homes. In addition, curriculum materials were examined 
and classroom instruction observed. 

Subjects and Locations: Delco 

Site #1, Delco Chassis, is a large, unionized (International Union 
of Electrical Workers, Local 509), electrical motor manufacturing 
plant with over 3600 employees located in Rochester, New York. 
Employees are enrolled in an education program jointly operated by 
union and management in conjunction with state and regional 
agencies that provide some funding and help in providing Instructors 
from the local school system. All learners were from the production 
teams. They participated in one of three types of class: 

1) Technical Preparation - a 6 week, 7 hours per day course 
designed to prepare employees for subsequent technical 
training, 

2) a GED preparation course which met in slightly varying 
time frames but generally 4 hours per week, and 

3) an ESL course which met for 8 hours per week. 

For the first of these, there was a control group, made up of workers 
who had not yet begun the Technical Preparation course. Each of 
these four groups consisted of 12 - 15 employees. In addition, a small 
control group (of 5) was available for the ESL group. 

The Technical Preparation course is designed to prepare 
learners for the mathematics, reading, oral communication and 
blueprint reading skills judged to be prerequisite for further technical 
training. Readings and activities are a mixture of some plant-specific 
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materials and carefully selected off-the-shelf materials related to 
course objectives* Activities in the reading component of the course 
include study skills exercises, reading rate exercises, and in-class 
activities designed to increase learner motivation to read* An 
instructors 1 manual of several hundred pages outlines course objectives 
and suggests materials and activities* Instructors are provided by the 
local school system, after screening by union and management 
representatives* Those who have been retained to teach the course 
have been able to demonstrate to these representatives that they can 
structure their teaching to meet course objectives and receive high 
instructor ratings from learners* 

The GED course involves a good deal of individualized study 
directed toward passing regularly scheduled GED tests* Learners used 
published test preparation materials as well as traditional school 
materials and work-book exercises from an extensive in-plant library* 
Use of individual learner folders, seat-work, some full-class discussion, 
and regular individual feed-back from experienced GED instructors 
characterizes how class time is spent* 

The ESL class is team taught by an experienced English as 
Second Language instructor and a Delco employee able to speak Italian 
(the first language of many but not most employees in the class)* 
Activities follow exercises in several published English as Second 
Language materials available in the Delco training center* Class time 
includes teacher demonstration of how to do language exercises, seat- 
work with both instructors providing individual feedback, and full- 
group discussion of correct answers and why answers are correct* 

Demographic data on the Technical Preparation class revealed 
most students to be in their late 20 s, averaging more than 12 years of 
education* The Test of Adult Basic Education is used by Delco to 
screen learners for placement in classes and provide some diagnostic 
information to instructors. Students in the Tech Prep class averaged 
near the top of the reading portion of the T.A.B.E. at between 11th- 
12th grade levels in ability before entering the class. 



64 

6<i 



STRUCTURE OF CURRENT STUDY 



A control group of employees, not yet enrolled in the Tech Prep 
course, was interviewed and tested* Analysis of demographic data 
revealed the control group to be slightly older, with more males and 
more years of plant experience than the class group . In most other 
ways, the two groups were similar, however- No significant difference 
was found for education levels or for the reading comprehension 
scores on a Cloze test* 



Demographic Information of Tech Prep and Control Groups 



Characteristic 


Tech Prep 


Control 


Age (mean in years)* 


27.9 


34.6 


Sex (M : F)* 


6 : 8 


11 : 1 


Service (mean in years)* 


5.9 


10.8 


Education (mean in years) 


12.28 


12.33 


Cloze Test Scores 


10.86 


9.58 



* Significantly different at the p< 0.05 level of significance 



Subjects and Locations: Cumberland 

Site #2, Cumberland Hardwoods of Sparta, Tennessee, is a non- 
unionized, rural wood processing plant with approximately 300 
employees. The plant produces several hardwood products for the 
furniture industry including drawer parts and components for kitchen 
cabinets. New technology and an ambitious quality assurance program 
have changed the nature of the work environment and of many 
traditional jobs. Cumberland has several small on-going training 
programs. Employees enrolled in the classes participating in this 
study were all from the plant floor. 
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One course at this site was entitled Communication and 
Collaboration - designed to train teams of employees involved in a 
given phase of the firm's operation* Several teams had already 
completed training in communications skills needed to work 
cooperatively as self-directed teams* The pilot study involved 
assessing two learning teams, each of 10 - 12 members, which the 
plant CEO described as the most difficult group of learners attempted 
so far* 

A second program at this site was an on-going GED course with 
6 students currently enrolled. The class is taught by an experienced 
Adult Basic Education instructor employed by the company. 
Instruction follows the demonstration and seat-work pattern 
described above for the Delco GED course. Earlier cycles of the GED 
course had allowed nearly 20 employees to complete the GED. 
However, because of the small number of current students and the fact 
that not all of them could be tested, insufficient data for useful analysis 
could be obtained for this group. 

The small size of this company prevented the formation of any 
control groups for either class. Also, because Cumberland has had an 
active and successful education program for more than three years, 
only a small fraction of employees had not yet passed through the 
small firm's training courses. 

Instruments 

Following a literature review for instruments and techniques 
employed to evaluate previous workplace literacy programs, a menu 
was constructed of available techniques for gathering data related to 
program impacts on productivity, learner gain and learner families. 

At each site, plant-gathered indices of productivity were 
surveyed and discussed until an agreed upon list could be developed 
for the site. In addition, supervisors participated in developing 
anchored rating scales on information processing tasks which were 
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plant specific (i.e., participation in meetings, doing quality assurance 
paperwork, etc.). These rating scales were used to rate learners 
before and after training. (See Appendix G for samples of these rating 
scales.) 

Interview, test and classroom questionnaire data were collected 
for each learner before and after each course, or at suitable intervals 
for on-going classes. Lytle's conceptual framework for changes in 
adult literacy (i.e.. Beliefs, Practices, Process and Plans - see Chapter 
3) was used as an organizational principle for the interview and 
questionnaire. Information was gathered on learners' beliefs about 
literacy in general and their own literacy in particular. In addition, 
interviews and questionnaires focused upon literacy practices, the 
literacy processes and abilities demonstrated with workplace literacy 
tasks, and learners' plans for 1, 5 and 10 years ahead. 

The instruments developed for the first phase involve a mixture 
of interview and questionnaire items which were to be used for all 
learners at all sites, and custom-designed tasks or job scenarios 
appropriate for particular sites and classes. For the Practices section 
of the questionnaire, site personnel added plant specific items to a 
more general list of reading material, which learners were to rate for 
difficulty. Questions related to literacy practices in work teams and in 
the plant were worded to reflect local language use. Questionnaire and 
focus group questions reflecting literacy practices with family 
members were also worded to reflect local use. For the Process 
section of the model, personnel at each site participated in analyzing 
workplace literacy tasks, and constructing Cloze tests and job scenario 
literacy tasks (i.e. reading plant newspapers or using job aids, forms, 
graphs, etc.) related to that workplace. (See Appendices A - D for 
sample instruments. The research rationale for construction of these 
instruments has been provided in Chapters 3 to 5.) 

A significant amount of instrument development occurred at the 
Delco site. Considerable time was saved at the Cumberland site by 
using the Delco instruments as models for modification or to stimulate 
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the thoughts of plant personnel about what might be useful tasks for 
the custom-designed portion of the assessment. 

Interview 

An interview protocol was devised to cover the four aspects of 
Lytle's model- Concerning their Beliefs, learners were asked to 
describe a literate person they knew at work and elsewhere, as well as 
how literate they saw themselves now f and as becoming in future* 

Concerning literacy Practices, learners were asked what reading 
and writing they had done recently, both at work and away from work. 
Literacy Process was tested using three different job-related items (i.e. 
a newsletter article, a graph and a job aid), which were all selected 
with the advice of the site coordinator* The subjects were asked to 
describe how they read or used the items, as well as answering 
questions about the specific contents. Finally, learners were asked 
about their future Plans, for 1, 5 and 10 years ahead, and how they saw 
reading and education as part of those plans. 

Questionnaire and Cloze Test 

A written questionnaire was also administered to participants 
during one of the first few class meetings, and again near the end of 
the course. Items dealt with the areas of literacy Beliefs and Practices, 
included a Cloze test based upon the local plant newspaper, and in 
addition contained questions about family literacy for those learners 
with children between the ages of 3 and 17- To complement the 
Beliefs questions in the interview (see above), the learners were asked 
to write down 4 or 5 words that described themselves as a reader and 
writer and to do the same for someone they saw as good at reading 
and writing. Further information about Practices was sought through a 
checklist of 20 possible types of reading material (e.g., books, signs, 
training manuals, pay stubs, charts, cartoons), for which the subjects 
were asked to rate, on a scale of 1 - 5, the difficulty they had in 
reading those of the items that they had read recently. They were also 
questioned about the frequency of literacy-related activities: how often, 
for example, they read a newspaper, made a shopping list or visited a 
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library, as well as how many books they owned. In relation to literacy 
at work, they were asked to respond, on a scale of 1 (very like me) to 
5 (very unlike me), to 10 statements such as "I just listen in 
meetings", "My ideas are discussed in meetings", "I read information 
when it is handed out", and "I have trouble reading information sent 
out by management". 

The questions about family literacy concentrated on literacy 
practices, particularly frequency of literacy activities: how often does 
the child look at books, read or ask to be read to, or visit a library; how 
often does the parent read to the child or help with reading; how 
many books has parent or child bought in the last year. 

For each site, the coordinator helped to select, from workplace 
materials, a suitable passage for use in a Cloze Test, in which every 
fifth word was left blank. These passages were of a page in length, 
with about 25 blanks to be filled in. 

Family Literacy Focus Group Interview 

At the Cumberland site, a group of learners with children were 
interviewed about literacy Beliefs and Practices in the home. They 
were asked, for example, why they thought some children did better 
at school than others and what kinds of literacy-related materials they 
had available for their children. Questions used in the focus group 
interviews reflect categories developed by Fitzgerald, Spiegel and 
Cunningham (1991) in assessing home and parental factors related to 
children's success in school. 

ESL Checklist 

Evaluation of English as Second Language proficiency is not 
easily done with paper and pencil measures since speaking, listening, 
reading and writing are all involved. Typically, teacher checklists of a 
wide variety of behaviors serve as a diagnostic record and instructional 
guide, and as an informal assessment of progress. 

Bronstein (1991) has developed an extensive workplace- 
specific ESL checklist entitled: Benchmarks and student learning 
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profile for the workplace ESL program of the Labor Ed ucation Center 
at Southeastern Mass. University . Instructors at Delco reviewed this 
checklist, selected items appropriate to their site, modified other 
items, and added a few items specific to their classes and workplace. 
This resulted in a list of competencies at three levels (Beginner, 
Intermediate, Advanced) dealing with such areas as following 
instructions, looking up information, and filling out forms. (See 
Appendix F for a sample of this modified checklist.) 

Class Observation Sheet 

Classroom observations were performed by research personnel 
and on-site coordinators using a class observation form developed by 
Mikulecky (1990) and utilized by Mikulecky and Philippi (1990) and 
Philippi (1991) in school and workplace settings (see below and in 
Appendix E). The form requires observers to describe instructor 
activities and student activities, and make comments about the nature 
of class activities on a timed basis. Notations are then shared with the 
instructor to corroborate the accuracy of what has been observed and 
to make note of purposes for some activities. 

Classroom Observation 

Time Student Activity Teacher Activity Comments 

0 

05 




Make note of time spent by students actually reading or doing things. Also note 
time learners spend listening to the instructor. When learners are in small 
groups or working individually should be mentioned. Special note should be 
made when the instructor or a student demonstrates how to do something. 
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Productivity Information 

Information on productivity needed to be custom selected for 
each worksite, though there was a small degree of overlap (i.e. 
attendance and safety records). In addition, each site participated in 
constructing plant specific supervisor ratings. 

Plant Gathered P roductivity Indicators 

Management at Delco Chassis gathers a significant amount of 
employee data related to achieving corporate goals. Researchers, 
working with management and union personnel, reviewed this data to 
select productivity indicators which could possibly be influenced by 
successful learning experiences in the workplace literacy program. 
Learner and control group pre- and post- data was collected on 
absenteeism, suggestions submitted, suggestions approved for awards, 
grievances submitted, discipline records, and workplace safety 
records. 

Supervisor ratings 

Extensive interviews were conducted with supervisors and 
workers to determine aspects of jobs that contributed to productivity 
and were related to communication, teamwork and paperwork skills. 
Ten aspects of job performance emerged from interview data at the 
Delco plant and ten aspects were also used at the Cumberland plant. 
Supervisors then provided examples of behaviors which separated top 
from middle from bottom performers on each scale. These behaviors 
were used to develop anchored rating scales for each of the 
productivity categories. Supervisors then rated each worker on these 
scales both before training and after training. (See Appendix G for 
samples of these rating scales.) 

Data Gathering Procedures 

Procedures for data gathering varied from instrument to 
instrument. Some were written directly by the learner or indirectly 
from the learner's comments, and others by the learner's teacher or 
supervisor, or by the researcher. 
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The interview and job scenarios were conducted by a researcher 
one-on-one with a learner* The researcher asked each question and 
made notes on the learner's responses, pausing long enough to obtain 
a considered answer, and using standard non-directive prompts and 
probes to elicit a more extensive response. The time taken for each 
individual interview was in the range 20 - 30 minutes. The Family 
Literacy Focus Group Interview was conducted in a similar fashion, and 
took about 10 - 15 minutes. 

The questionnaire and cloze test were administered by the 
teacher during the class period. Each learner filled out the answers 
individually, with the teacher available to explain or clarify items when 
the learner was unsure what to do. 

The ESL checklist was completed by the teacher of each student 
in an ESL class, and the class observation sheet was completed by a 
researcher while a class was in progress. 

In some cases, supervisor productivity ratings were completed 
in conjunction with a researcher. At other times, rating forms needed 
to be left with supervisors. Hiis divergence in procedure may have 
contributed to difficulties at the Delco plant in obtaining consistent 
supervisor ratings. (See Chapter 7 for more details on this.) 

Data Analysis Techniques 

Data analysis was of two types. Cloze test scores and quantifiable 
questionnaire and interview responses were recorded and analysed 
statistically. Responses to open-ended interview questions were 
recorded, and then methods of analysis were developed to fit the 
nature of the responses. 

For some open-ended interview questions, categories of 
responses were allowed to emerge from data. These categories were 
then used to label subject comments. When category refinement 
allowed for acceptable levels of inter-rater agreement (90% or 
higher), category responses were recorded and statistically analysed. 
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For other open-ended interview questions, a holistic comparison was 
made between pre-test and post-test responses, and the change was 
rated as positive, neutral or negative. As with the category schemes, 
the criteria for assessing this change emerged from data, and the 
application of the scheme was subject to the same levels of acceptable 
inter-rater agreement. 

Examples of both category and holistic rating schemes arose in 
connection with the interview question: 

"How literate do you consider yourself to be? 
What makes you think so?" 

Responses to this open-ended question nearly always included 
spontaneously some kind of self-rating, using words such as "average", 
"very literate", "below average", "poor". These were categorized from 
lowest to highest on a scale of 1 - 5, to produce a score for each self- 
rating. In addition, a holistic rating was applied to the full response, 
in which change from pre-test to post-test was judged as positive, 
neutral or negative according to the subject's reported self-image and 
the reasons given for it. 

For any of the responses which resulted in numerical scores, 
statistical tests were applied to the set of scores for each group of 
subjects. Pre- and post-assessments were compared for the 
individuals in a class using a paired-sample t-test, in order to detect 
gains brought about by the program. Where a class had a control 
group, the changes for the two groups were compared using a two- 
sample t-test. In addition, for the holistic change scores, the 
allocation of values +1, 0 and -1 to positive, neutral and negative 
allowed the use of a one-sample t-test to find if the changes were 
significantly different from 0. ^In all cases, as the tests were of "no 
difference" versus "improvement", the statistical tests were one-tailed. 

Conclusion 

This study's objective was to develop an evaluation model that 
can be used with most workplace literacy programs. A pilot evaluation 
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was conducted at two very different workplaces, where data was 
obtained on productivity, learner literacy attributes and learners' 
families. This data was gathered, before and after each course, using 
learner interviews and questionnaires, company records, and 
supervisor ratings of employees. Analysis of the data included coding, 
scoring and categorizing items, and applying statistical tests to detect 
improvements that had taken place during the time learners were in 
class. 



Chapter 7 
Results of Current Study 



Overview 

The main purpose of this chapter is to indicate which of the 
evaluation techniques employed have been most successful in 
detecting pre /post program changes in learners and their families, 
and in employer objectives. This will be illustrated using examples 
from 

• the Technical Preparation class at Delco 
(often contrasting it with its control group); 

• the GED class at Delco; 

• the ESL class at Delco 

(making some comparisons with its small control group); 

• the Communications and Collaboration class at Cumberland 

(to a lesser extent, as less data gathering could be done there). 

Pre-test and post-test results were compared statistically and 
analytically for each class studied, on each aspect of measurement 
used: learner beliefs, practices, processes and abilities, and plans, 
family literacy and employer objectives. Program impact on learners 
from the various Delco and Cumberland classes is summarized below. 
CTabular data are available in Appendix H.) 

These results provide the principal basis for revising some 
aspects of the evaluation instruments and retaining others, as 
described in Chapter 8. 

Learner Literacy 

Changes in Beliefs 

• View of a literate person - no change 

• View of self as literate - significant gain for Technical 
Preparation, but not for control 



EVALUATING THE IMPACT OF WORKPLACE LITERACY PROGRAMS 



Changes in Practices 

• Reading and writing at work - significant gain for ESL, but not 
for control 

• Participation in meetings - significant gain for Technical 
Preparation, but not for control 

• Asking questions at work - significant gain for ESL (but not for 
control) and significant gain for Cumberland 

• Reading and writing away from work - significant gain for 
Technical Preparation, but not for control 

• Range of reading - significant gain for GED and ESL, but not 
for ESL control 

Changes in Reported Reading Process and in Reading Comprehension 

• Job-related cloze test - significant gain for Technical 
Preparation, but not for control 

• Prose reading process - significant gain in responses 
(particularly topics mentioned) for GED and ESL, but not 
for ESL control 

• Job scenario questions - significant gains for all Delco classes 
on questions of various difficulties 

• Use of job aids - significant gain for ESL, but not for control 

Changes in Plans 

• Plans for 1 and 5 years - significant gain in focus and literacy 
goals for Technical Preparation, but not for control 

• Reading and education in plans - significant gain for ESL, but 
not for control 

Family literacy 

• No change for GED and ESL; number of parents in other 
groups too small for statistical analysis 

Employer Objectives 

• Attendance - no significant change for Delco groups 

• Safety, suggestions, etc - numbers too small for statistical 
analysis 

• Supervisor ratings - significant gain for Cumberland 
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Changes m Learner Literacy 

Learner beliefs about their own literacy and about what it means to be 
a literate person were assessed with both questionnaire items and 
open-ended interview questions. 

Beliefs 

Subjects' views of what constitutes a literate person did not 
change significantly from pre-test to post-test, but it was noteworthy 
that their comments ranged quite widely, beyond the area of reading 
and writing, to include mention of broad-based intellectual and social 
qualities and the sorts of things literate persons were able to do. 
Examples of types of comments used to describe a literate person 
included mentioning such attributes as "college education", "knows a 
lot", "experienced", "has a better job", and such abilities as "well- 
organized", "competent", "helpful", "concerned", "good at solving 
problems". There were also more expected comments like "reads all 
the time", "understand what they read", and "writes well". 

In response to the interview question "How literate do you 
consider yourself to be?", the Technical Preparation group showed a 
statistically significant improvement, and the ESL group also showed 
some numerical improvement. This was measured in two ways. 
Responses to this open-ended question nearly always included 
spontaneously some kind of self-rating, using words such as "poor", 
"average", and "very literate", which was scored on a scale of 1 - 5. 
(The three examples just given would score 1, 3, and 5, respectively.) 
A holistic rating was also applied to the full response, in which change 
from pre-test to post-test was judged by the reported self-image and 
the reasons given for it. These changes were rated as negative, zero or 
positive. For example, one learner's responses that received a positive 
rating were: 

Pre: "Not very literate- not much education." 

Post: "I'm average. I'm not stupid. I have common sense and 
can read and write." 



EVALUATING THE IMPACT OF WORKPLACE LITERACY PROGRAMS 

Another made gains at an apparently different level: 

Pre: "Fair or average- a bit above. I understand some 

words, but others I don't. I'm not sure if it's literacy 
or memory." 

Post: "I'm more literate than I was before class - I understand 
more. I'm getting more interested in fiction, and fact. I 
look up words in the dictionary and thesaurus." 

Pre/post changes for the self-rating and the holistic scores were 
statistically significant at the p<0.02 and p<0.01 levels for the 
Technical Preparation group. Control group scores showed no 
significant change. 

A question about literacy beliefs, corresponding to that in the 
interview, was included in the written questionnaire. It asked the 
subjects to write down several words that described themselves as a 
reader and writer. This format did not produce the same result. 
Here, pre /post variation was due more to the number of words written 
down than to any change in the nature of the response. This 
illustrates an advantage that the interview had over the questionnaire 
in gathering richer data - a difference to which we shall return in later 
sections. 

Thus it appears that access to learners' beliefs about literacy is 
more easily obtained through interviews than questionnaires, and their 
beliefs about themselves are more likely to change than their beliefs 
about others. So it is probably most useful to see the opening question 
on the interview protocol in Appendix A ("Describe a person you know 
who is good at reading and writing") more as a warm-up question to 
start the learner thinking about literacy than as one likely to produce 
evidence of change brought about by a program. The later questions, 
asking how good at reading and writing subjects consider themselves 
now, and likely to become in future, appear more responsive to change 
between pre- and post-assessment, and provide a useful measure of 
the effect of a program on a learner's beliefs about literacy. 



7 8 78 



RESULTS OF CURRENT STUDY 



Practices 

Learners were questioned about their literacy practices, both at 
work and at home. Concerning work-related activities, they were 
asked in the interview to describe the kinds of reading and writing 
that their work had involved during the past week, and in the 
questionnaire to rate on a scale from 1 (very like me) to 5 (very unlike 
me) a number of statements relating to contributions in meetings and 
the reading of work-related materials. 

The interview responses were assessed by a count of items 
mentioned, and by holistic pre/post change judged by the breadth, 
frequency and difficulty of the reading mentioned. In general, these 
measures showed pre/post gains. For the ESL class (but not for its 
control group), the changes were statistically significant (p<0.05 and 
p<0.01 for the two measures). The nature of the gains is illustrated by 
these sample responses. 

Pre: "Newspaper - during break and lunch." 

Post: "Read check sheets for parts, suggestions, bulletins, 
QUILS, monthly quality paper. Writing: Check off on 
sheet." 

Pre: "Nothing really - just put parts on the line." 

Post: "Bulletin at work. I can really read it now. The 

information is important. I read the magazine at work, 
also; it's new." 

■4 

The one exception to this pattern of pre/post gains for workplace 
reading was the Technical Preparation group, which was in class full- 
time and therefore had not been doing their normal work for the 
duration of the course. For such full-time classes, it would be better to 
conduct the post-interviews a few weeks after subjects' return to 
normal work, in order to register any changes in work-related reading 
behavior resulting from their training. 

Learner self-ratings on the statements about meetings and work- 
related reading showed very little pre/post change overall, but a few 
aspects are noteworthy. For the Technical Preparation group, two 
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items showed significant increase (p<0.05 for both): those concerned 
with talking in meetings and with having one's ideas discussed in 
meetings. For the ESL group, but not its control, the following item 
showed a significant increase (p<0.05): 

When you need to know something at work, you usually ask someone about it. 
very like me 1 2 3 4 5 veiy unlike me 

This reflects an emphasis on oral work in the ESL class and shows a 
gain in confidence by these workers. This item also showed 
significant gains (p<0.02) for the Cumberland Communications and 
Collaboration class, which put much emphasis on working 
cooperatively with other workers. Thus, in both cases, skills dealt 
with in class produced changes in workplace behavior. 

Turning now to literacy activities away from work, the learners 
were asked in the interview to describe the reading and writing that 
they did away from work, and in the questionnaire to rate themselves 
on their frequency of several literacy-related activities and their 
ownership of reading materials. 

The interview responses were assessed by a count of items 
mentioned, and by holistic pre/post change judged by the breadth, 
frequency and difficulty of the reading mentioned (as described above 
for workplace reading). This showed statistically significant increases 
for the Technical Preparation class in both measures: the count of 
items and the holistic rating (p<0.02 and p<0.005); the control group 
showed no such increases. The GED group also registered some 
numerical gains, but the ESL group did not. The lack of change in 
reading and writing for the ESL class may be due to the emphasis on 
oral work already mentioned, or to their separate uses of English at 
work and their native language outside the workplace. 

For the section of the questionnaire about frequency of literacy- 
related activities away from work and ownership of reading materials, 
no items showed significant change and it may be that changes in 
these areas of behavior and ownership are slow to take effect, 
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requiring more than the few weeks of time available between pre- and 
post-testing. In their post-interviews, a number of the learners 
expressed their positive intentions in such areas, but the stimulus 
from attending the course was then only beginning to produce 
changes in behavior. Comments of this kind included: 

"I have to do more reading for my daughter - especially now that 
I have more incentive from this class. It's like a spark." 

"After this course, I'm going back to night school. I'm really 
impressed with this class." 

Though the Technical Preparation class did not improve 
significantly in home literacy behaviors, their improvement scores 
were significantly better (p<0.01) than those of the control group, 
which actually reported less home reading in the post-test. Perhaps 
this reflects a baseline of less general reading in summer (when the 
post-tests were conducted), because of the availability of other leisure 
activities. 

In the questionnaire, learners were also presented with a list of 
20 types of reading- some general (e.g., newspapers, books, bills) and 
some plant specific (e.g., Delco Doings , suggestion forms, route 
sheets, paycheck stubs). They were asked to rate each on a scale 
from 1 (easy) to 5 (hard), also indicating which of the types they had 
read in the last month. This revealed a statistically wider range of 
reading in the post-assessment for the GED and ESL groups (p<0.01 
and p<0.002). The ESL control group did not show such gains. 

Few of the individual items showed significant change, but over 
half (about 11 of the 20 items, averaged over the 4 classes) were rated 
by learners to have greater perceived difficulty. This may mean that 
learners were being more realistic in the post-assessment, after 
greater exposure to reading generally, or just that they were unable to 
apply the scale consistently over the time gap between pre- and post- 
test. For those not accustomed to using scoring schemes, there may 
be a problem in such assessment, particulary self-assessment. (See 
also supervisor ratings in the Productivity section.) One aspect of this 
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question's wording may also have contributed to the difficulty: if the 
learners had been asked first to indicate which of the types they had 
read recently and then to rate only those items, it is possible that 
more consistency might have been obtained. 

These difficulties point up once again the inflexibility of a 
questionnaire and its dependence on the abilities and willingness of 
the person filling it out, compared with an interview in which the 
interviewer can explain a question and probe for further information 
to clarify the learner's intentions. 

Process and Ability 

In the interview, workers were asked to respond to both 
process and content questions on a plant newspaper article, a 
moderately complex graph, and a job instruction sheet. They were 
also given a Cloze test constructed from plant reading material. 

The Cloze test used at Delco came from a plant newspaper 
article. The Technical Preparation class made statistically significant 
pre/post gains (p<0.02), while its control group did not. Nor did the 
GED and ESL groups, but the reason here appeared to be that the 
reading passage was too difficult for them. These two groups had 
mean scores of about 7 (out of 23 blanks to be filled), which indicates 
a frustrational reading level, but the Technical Preparation class and 
its control group had means of 10 or 11, well above the frustrational 
level. (50% replacement indicates an independent reading level, 
while below 35% indicates a fiustrational reading level.) Given this 
range of reading ability, it would have been better to have had available 
two (or even three) Cloze tests of different difficulty levels, to be used 
as appropriate. 

The Cumberland Cloze test used the plant safety rules, and here 
the mean score was 14 (of 25 blanks), showing that the test was well 
within the reading ability of those taking it. However, no pre/post 
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comparison was available at Cumberland, since the on-site coordinator 
managed only a single administration of the test. 

A portion of the interview involved responding to a newspaper 
article, a graph and an instruction sheet in job-related scenarios. 
Learners tended to answer correctly the simpler fact-level content 
questions on the pre-test, so no gain in tliis area was apparent on the 
post-test. Significant gains were demonstrated on more complex 
items, which often called for the use of inference and interpretation. 
For example, the Technical Preparation group showed significant 
gains on the most difficult article and graph questions (e.g. "What 
happened to the inventory value in August and September?", which 
required learners to describe a trend), while the ESL group did so on 
those of medium difficulty (e.g. "What is the inventory value for the 
week of August 19?", which required reading from between two scale 
values). Since the levels of competence of the different groups (and 
individuals) varied, a greater range of difficulty in the sets of questions 
would have allowed improvements to show at an appropriate level. 
This would also have been assisted by a wider range of item types in 
each section, from the simple factual to more difficult interpretation 
questions. 

When asked to read a plant newspaper article and describe how 
they read it (i.e. the processes they were using), learners* responses 
covered two main areas: reading strategies and topics of interest. 
Strategies included skimming, starting with headings and bold print, 
and reading the first and last paragraphs. Topics of interest included 
the products manufactured by competitor companies and the wages 
that they paid. Responses to this process question (reproduced 
below) included the following examples: 

Describe what you would look at. What would you be 
thinking about? How would you go about reading this story? 

Pre: "Check each heading and decide whether to go further." 

Post: "Read the headings, get ideas about the companies, skim, 
know what they make, and know their customers and 
competitors." 
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Pre: "Read the first paragraph, read the headlines and bold 
print, and pick out what hits my attention." 

Post: "Read the dark print first, then break it down from there 
and read what gets my attention. Also, I'd find out what 
it's about and what they are telling in it. Then, I'd read it 
in depth." 

Pre: 'Title, first paragraph, through the whole thing - analyze it." 

Post: "Title, subject ("Delco Doing; "), and function and operation 
of companies. I would look at Asia and Europe (competitive 
markets) to see how their prices are lower and higher." 

Pre: "First the title, then I'd read from beginning to end." 

Post: "Start at the beginning. Look at the areas and read all the 
way through. I'd also read about how Delco is trying to 
compete, its main customers and the percentage of 
wages and benefits. I'd read "Delco Doings" to find out 
about Delco's further needs. Reading these things makes 
you familiar with other companies." 

These responses were analysed by counting the number of 
separate items mentioned by the interviewee. For all the Delco 
classes, the total number of responses and the number of topics 
mentioned increased numerically, and these were statistically 
significant for the GED and ESL groups (p<0.005 in both cases); the 
control groups showed no gains in these areas. The increase in the 
number of topics that the learners mentioned shows a greater ability 
to make connections between what they read and their own 
knowledge, as well as showing a growth in confidence arising from 
their time in class. For the most part, increased discussion of 
strategies included comments one would expect from more 
sophisticated readers. 

In connection with the job instruction sheet, learners were 
asked about their use of such job aids, how long it took them to read 
one, and how difficult they found it. The only case of a statistically 
significant gain was for the ESL class (but not for its control group) in 
reply to a question about how likely they were to use a job aid. It 
appears that ESL learners' confidence in approaching job-related 
reading had been increased by their attendance of a class. Other 
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questions that involved self-reporting of reading skills were not 
successful, because of the interviewees' inability to gauge their own 
capacities* A question about the length of time it took to read a job 
aid produced a wide spectrum of answers: from 1-2 minutes up to a 
week. Responses at the bottom end of this range were clearly 
unrealistic: just one of the content questions tended to take more 
than 2 minutes. And responses like a day or a week seem to refer to 
the length of time needed, not only to read the job aid, but also to 
learnHhfi job it relates to. Ambiguity in connection with such items 
may well preclude their effectiveness. 

In all of the job scenarios (newspaper article, graph and 
instruction sheet), the content questions showed a considerable 
variation in response, both between groups and between individuals. 
To accommodate this, the set of questions for any one section needs 
to range in difficulty and in nature (involving fact, inference and 
application), so that there is room for improvement from pre- to post- 
test, at some level, for all those interviewed. 

Plans 

In the interview, learners were asked about their plans for the 
future: 1, 5 and 10 years ahead, and to explain how reading and 
education formed part of those plans. Assessed on how definite and 
detailed the plans were, the Tech Prep students showed significant 
pre/post improvements for 1 and 5 years (p<0.02 and p<0.05); these 
did not occur with the control group. The ESL class showed a 
significant increase in references to reading and education as part of 
their plans (p<0.005), which was not repeated for its control group. 

Responses to planning questions ranged from learners 
mentioning their prospects for advancement in the company, or out 
of it, to their intentions regarding marriage, children, housing and 
retirement. The following are typical responses to the question: 

What are your plans for the next year? 

Pre: "Finish degree." 
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Post: "Getting married. Going to school - 4 nights a week in the 
fan." 

Pre: "Have another child, learn new jobs in the same 
department, and get a new car. 

Post: "Have another child, lose weight, take some course (I don't 
know what kind), and probably finish a degree in retail." 

What are your plans for the next five years? 

Pre: "Apprenticeship completed." 

Post: "Have electrical apprenticeship, have kids - maybe, (and) 
perhaps buy another house." 

Pre: "Go through the apprenticeship. This will take up a big 
amount of time." 

Post: "Be done with the apprenticeship and definitely move up 
and off the assembly line." 

Pre: "Ending an apprenticeship - journeyman or greater 
position." 

Post: "I'd like to have a combination of school and owning my 
business." 

The connection with literacy was made more explicit through 
the follow-up question about the role of reading and education in their 
plans, as these comments from the post-interview reveal in response 
to the prompt: 

Explain how you see reading and education in these plans. 

"Reading helps with everything. As you grow you learn. I want 
my life to grow." 

"I need to develop and build confidence - do more reading - 
that's definitely important. Get my kids to do more of it." 

"Get a better job by taking more classes. Help my kids read 
more - help them in school." 

"If you can't read, you can't trouble-shoot the machines. " 

"The more I learn, the easier it. is to make suggestions about 
things and to apply for better positions." 



ERIC 



RESULTS OF CURRENT STUDY 



"I feel better after being in here and I want to learn more. I have 
to read a book on game for hunting. To retire, I need to 
read about benefits." 

Overall, the learners were very positive about their experience in 
classes and saw them as opening new doors, both for further education 
and for a life of greater opportunity. 

Changes in family literacy 

Measures of family literacy mainly involve information about how 
parents interact with their children in literacy-related activities. In 
addition, some of the parents* reading behavior away from work 
reflects upon changes in family literacy. 

Learners with children between the ages of 3 and 17 were asked 
in the questionnaire about the frequency of such activities as reading 
to their child, helping the child with reading, and buying books for 
the child. Also, they were asked how often the child read alone, and 
what kinds of books or other materials (if any) the child borrowed 
from a library. 

These questions revealed no statistically significant changes for 
the GED and ESL groups, the only ones for which the number of 
parents was large enough to draw any conclusions. These groups each 
contained 12 parents of children in the relevant age range, while the 
Technical Preparation class and the Cumberland group contained only 
4 each. 

However, the responses to these questions on family literacy did 
show slight overall gains, even though not statistically significant ones. 
It may be that the time between pre- and post-tests was not enough 
for any changes brought about by the classes to have much effect. It 
may also be that a larger sample size would have revealed the trend 
for improvement to be statistically significant. In addition, there is 
some evidence in the oral interviews about reading practices to 
indicate a movement towards more literacy activity for oneself and 
one's children. During post-test interviews, class members were 
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more likely to report newspaper, magazine and novel reading. Two 
parents, who had not previously mentioned reading to children , 
during post interviews mentioned "reading to my child" or "reading a 
children's book to my son." Another reported reading child care 
books and magazines on parenting. These had not been present 
during pre-interviews. One class member commented, "I definitely 
read a lot more since I started taking this course." 

Changes in Meeting Employer Objectives 

In relation to worker productivity, measures used at Delco were 
attendance, safety records, suggestions submitted, suggestions 
approved for awards, grievances submitted, and discipline records. In 
addition, each site participated in constructing plant-specific 
supervisor ratings. 

No significant changes occurred in learner attendance, but it 
was noted that, due to small sample size, the absence figures of a few 
individuals could affect the total quite markedly. For example, in the 
Technical Preparation group, half the absences in the post-training 
period are attributable to three employees. So, with samples this 
small, extreme caution should be used. 

Other measures such as productivity suggestions and accident 
records involved numbers too small for statistical testing. 
Suggestions were made by only a few members of each group during 
the periods concerned and these followed no apparent pattern. 
Accidents were even rarer; for example, no Technical Preparation or 
control group member had an accident during the six weeks prior to 
training. Such figures do not allow statistical analysis. 

At both Delco and Cumberland, supervisor ratings were devised 
to measure aspects of jobs that contributed to productivity and were 
related to communication, teamwork and paperwork skills. Extensive 
interviews were conducted to determine relevant skills and, at each 
site, ten aspects of job performance emerged from interview data. 
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Supervisors then provided examples of behaviors which separated top 
from middle from bottom performers on each scale. These behaviors 
were used to develop anchored rating scales for each of the 
productivity categories. Supervisors then rated each worker on these 
scales both before and after training. 

At Delco, the supervisor ratings of the workers' job-related skills 
produced some anomalies that cast doubt on the consistency of the 
ratings from pre-test to post-test. Even though supervisors 
participated in scale development, some seemed to rate some 
workers exactly the same on all scales. Some of the ratings appeared 
to be carelessly done. Even with certain items and individuals 
removed to correct fcr this, no change was apparent. It may be that 
supervisors need more training or instruction before doing ratings, or 
that a time period of six weeks may be too short to register 
improvements. 

At Cumberland, however, all ten aspects of the assessment 
scheme used there showed significant improvements (p<0.0001), over 
the 1 1 weeks of the classes. Here, just two individuals made the 
assessments and made them for the same workers in both pre- and 
post-tests, whereas at Delco up to four supervisors assessed the 
members of each group and there had been some personnel changes 
between pre- and post-test. Also, the Cumberland assessors had 
slightly more education and were not shop-floor supervisors, as at 
Delco; they may have had more experience in making judgments and 
ratings. 

Another factor that may have contributed to the Cumberland 
results is the choice of assessment categories. These were very 
closely related to the objectives of the Communications and 
Collaboration course, covering such items as communication skills, 
problem-solving ability and conflict resolution. The Delco assessment 
referred mainly to specific job skills such as machine setting and 
record-keeping, but Delco courses were of a more diffuse job training 
nature, not relating directly to these skills. This tends to confirm the 
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notion that learners gain knowledge and skills only in the areas that 
are taught. 

Conclusion 

The instruments used in this study to measure the impact of 
literacy programs varied in their success at detecting pre /post 
changes in learner literacy, family literacy and employer objectives* 
The main results and some observations about assessment utility 
follow. 

Learner literacy 

Beliefs 

• Learners reported improvements in their view of themselves 
as literate, but not in their view of a literate person. 

• Interview questions were more successful than questionnaire 
items in detecting changes in self-image. 

Practices 

• Reading practices at work improved in areas that related to 
the class attended. 

• Full-time classes need to be post-tested some time after 
learners return to normal work, so that changes in work- 
related reading can take effect. 

• Reading practices away from work improved for classes where 
home reading had been encouraged. 

• Interviews were more sensitive to changes in reading 
practices than were questionnaires. 

• Self-assessment of reading difficulty produced some 
inconsistencies that cast doubt on this questionnaire 
section. 

Process and Ability 

• Cloze test scores improved only when the passage was at 
an appropriate reading level, suggesting a need for several 
different test passages. 

• Answers to process questions on job-related reading materials 
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showed improvements in reading strategies and topic 
connections. 

• Answers to content questions on Job-related reading materials 
showed improvements at various levels for different 
classes, suggesting a need for a range of difficulty and type 

in the questions. 

Plans 

• Learners were generally more definite and detailed in their 
plans for the future after attending classes. 



Family literacy 

• Questionnaire items showed slight gains in some areas, but the 
time may have been too short for s^riflcant improvements 

to occur. 

Employer Objectives 

• Attendance showed no significant changes; with small 
groups, there is a problem of a few individuals' absence 
distorting totals. 

• Safety, suggestions, etc were too infrequent for analysis. 

• Supervisor ratings showed significant gains when areas 
covered related to the class, as well as the job. 

• Consistent supervisor ratings are difficult to obtain across 
several supervisors and when personnel change from pre- 
to post-test. Education and experience levels of 
supervisors may also be a factor in obtaining consistency. 
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Chapter 8 
Discussion and Implications 



Overview 

This pilot study has shown that it is possible to perform a broad- 
scale assessment of workplace literacy programs, in order to measure 
the impact on learners, their families and their productivity. The 
results of the study demonstrate some improvement in each aspect of 
the assessment model. However, gains appear to be limited to what is 
taught; there is very little transfer to other areas not addressed by 
instruction. 

Learner change was measured in the areas of Beliefs, Plans, 
Practices, and Processes and Abilities. Where these formed a part of 
class instruction, learners made gains in the following areas: 

• their literacy self-image; 

• their ability to articulate plans; 

• the amount and range of literacy activity 
both at work and away from work; 

• reading strategies and comprehension. 

Classes did not address directly issues of family literacy, and little 
change was evident in this area. Productivity measures proved, on the 
whole, to be unsatisfactory for the small numbers of learners studied, 
although supervisor ratings showed increases when areas assessed 
were closely related to instruction and company goals. 

The evaluation model itself was also under scrutiny in this 
project. Several points of interest have arisen from the pilot 
assessment. 

1) Questionnaires, although time-efficient, seem to be less effective 
than interviews in gathering accurate information. 
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2) Because of the range of learner abilities, workplace scenarios need 
to include questions of a variety of difficulties; cloze tests of 
varying difficulty may also be necessary. 

3) It would be desirable to have direct measures of learner 
productivity, as well as more reliable ways of obtaining supervisor 
ratings. 

Principal achievements 

A good deal has been learned from this pilot assessment. 
Initially, the pilot study has demonstrated that it is possible to 
perform a broad-scale assessment of workplace literacy programs 
using learner interviews, tests, and questionnaires within reasonable 
time-frames (i.e. 20-30 minutes of interviews and 10-15 minutes of 
tests and questionnaires, both before and again after instruction), as 
well as company records and supervisor rating scales. During 
subsequent studies, it will be determined if the assessment model can 
be transferred to additional workplace literacy programs with a 
minimum of technical assistance. 

Secondarily, results from the assessment provide indications of 
what effective workplace literacy programs can accomplish and may 
not be able to accomplish. Discussion of these results will reflect and 
substantiate two major generalizations. These generalizations are: 

1) Workplace literacy program instruction is able to demonstrate 
positive improvement in each area of the assessment model (i.e. 
beliefs, practices, processes and abilities, plans, productivity, 
and family literacy). 

2) Gains seem to be limited to areas directly addressed by instruction 
(i.e. programs and classes accomplished gains only in areas 
where there was direct instructional activity). No clear carry- 
over or transfer to other areas is apparent in evaluation results. 

impact of Programs and Link to Instruction 

Data were gathered for the learners in a range of classes: 
Technical Preparation, GED, ESL, and Communication and 
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Collaboration. Program Impact results will be summarized and 
discussed in direct relation to the types of instruction in classes 
where gains were made. The types of instruction in classes not 
demonstrating gains in particular areas of the assessment model will 
also be discussed for comparison purposes. 

Beliefs About Self as literate 

Changing adult learners' beliefs about their own literacy abilities 
is important for several reasons. An adult with a negative self- 
impression of his or her own ability is not likely to attempt literacy 
away from the supportive environment of a classroom and nurturing 
instructor. Significant growth in literacy abilities requires hundreds of 
hours of practice — more than most programs can ever provide in 
class time. For this reason, it is important that learners become more 
independent by developing more positive self-beliefs about their 
literacy abilities. They need to see themselves as being capable of 
attempting more literacy and practicing more on their own, as 
opposed to avoiding literacy tasks and literacy practice. (Incidentally 
it is important that learner beliefs about their own literacy are also 
accurate or learners will feel betrayed when they realize they have 
been wrong, and that betrayal can lead to abandoning altogether much 
of what has been learned in classes). 

Data about learner literacy beliefs were collected in all classes. 
Except in the GED class, learners demonstrated improved views of 
themselves as literates. This was mainly revealed during interviews 
through more positive self-descriptions and self-assessments. 

In the Tech Prep class, learners were regularly able to monitor 
their own progress on reading comprehension and reading rate 
through class tests and discussions. In addition, class discussion time 
during seven-hour learning days often addressed future learning plans 
and why the skills students were mastering would be of use in future 
training. 

The ESL class, similarly, used class discussion both to provide 
English practice and to highlight the relevance of what was being 
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learned to future use. Learners were asked to share, in journals and 
later oral discussion, personal accomplishments in written and oral 
English. This served both as an instructional tool to improve language 
use and also as a feedback mechanism for reinforcing learners 1 views 
about their own growing language and literacy competence. 

The Communication and Collaboration class revolved 
substantially around the concept of joint and personal goal setting, 
planning for accomplishing goals, and monitoring effectiveness. Some 
goals related to direct job performance, but a substantial number 
related to improving individual communication abilities. Considerable 
time was expended on both individual and group monitoring of gains. 
An apparent result was expanded and improved beliefs about learners 1 
own literacy abilities. 

The GED class demonstrated no gains in the area of improved 
learners 1 beliefs about their own literacy abilities. The structure of 
classes did not lend itself to either substantial instructor feedback, 
group feedback, or individual monitoring in this area. Most work was 
individualized and directly related to completing practice exercises 
for the GED tests. Interviews with learners often indicated a 
workmanlike attitude toward how many exercises they had gone 
through with little sense of improved individual abilities beyond the 
class. Instruction did not focus on learners internalizing a sense of 
expanded personal abilities, and assessment did not reveal such 
changes in belief to have occurred. It is important to note that other 
assessment measures indicated that GED students were learning. 
They had actually improved in some literacy abilities and practices. 
The significant factor here is that tittle class time was directed toward 
identifying and reinforcing growth in this area and concurrently no 
change in individual beliefs about changed literacy abilities was 
demonstrated. 

Plans 

Interview questions about future plans and the relationship of 
literacy to those plans were asked of learners in the Tech Prep, GED, 
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and ESL classes. In the Tech Prep class, which was designed as a 
prelude to further training, a good deal of time was spent addressing 
study skills and the demands of future training. In this class, learners' 
post-class interviews revealed plans which were articulated with more 
focus and detail than had been true of pre-class interviews. A similar 
pattern occurred in the ESL class which sometimes used discussions 
of learners' futures as an activity for improving the use of oral English. 
Learners in the GED class, who primarily focused on passing the GED 
test and were involved mainly with individual seat-work, demonstrated 
no measurable change in the clarity or focus of their plans and made 
no greater mention of education and literacy as parts of future plans. 

Literacy Practices 

Changes in the amount and types of literacy practices used by 
learners were assessed by a combination of interview questions and 
questionnaire checklists and rating scales. The same pattern of gains 
directly related to classroom focus areas was revealed in assessment 
results in all classes. 

Tae Tech Prep classes emphasized increased reading habits for 
learners. One instructor even took learners to the library, read 
portions of books and magazines aloud in class, and emphasized 
improved literacy habits. In the Tech Prep classes, even though the 
course title implies workplace applications, learners demonstrated 
increased reading and writing at home in post-interview questions 
compared to the literacy behaviors of the control group. Employees 
from only a few work-teams were enrolled in these classes and 
learners participated in a significant amount of group work in class 
with team members. Learners also reported statistically significant 
increases in willingness to offer ideas and discuss them in quality 
assurance meetings. 

The ESL learners spent some class time on workplace literacy 
habits and practices by reading bulletin board items, newsletters, and 
job materials in class. Significant gains in pre/post literacy practices 
at work were reported by ESL learners. These gains were also 
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significantly greater than those of a small ESL control group who 
served as an indicator of changes resulting from simply living for a 
comparable period of time in an English-speaking environment. Little 
class time was allocated to home literacy activities and no significant 
improvement in this area was noted- Oral discussion of class exercises 
(such as how to convert statements to questions) took a significant 
amount of class time. The heavy emphasis on oral usage and asking 
questions led to significantly higher ratings for asking others for 
information in the workplace. 

Learners in the Communication and Collaboration class also 
participated in a significant amount of group activity. They, too, 
showed gains on items involving question-asking and communicating 
in the workplace. Lack of post-assessment interviews prevented more 
extensive examination of changes in other literacy practices. 

The GED class, which did not emphasize literacy practices at 
home or at work, did not show gains in these areas on interview 
items. Questionnaire checklists, however, revealed a statistically 
significant tendency for learners to report attempting a wider variety 
of reading and writing. 

An interesting phenomenon was noted for GED and ESL 
learners' responses on these checklist items. In addition to indicating 
what types of materials they had read recently, learners were asked to 
indicate the difficulty they had reading these materials. Learners in 
both classes indicated that materials were harder to read at the end of 
the classes than they had been at the beginning of the classes. This 
suggests that low literates without much experience with reading may 
initially over-estimate their own abilities. It further suggests that 
instructors should not simply propose extra reading without providing 
support. When low literates realize that even simple materials are 
more difficult than they anticipated, they may become discouraged and 
retreat from reading them. 
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Reading Processes and Abilities 

The most psychometrically rigorous measure of reading ability 
used in this study was the Cloze reading test. The Tech Prep class, 
which spent the most time in reading practice, was the only group to 
demonstrate statistically significant gain on this measure. GED and 
ESL learners, who found the newsletter story used in the Cloze test to 
be at their frustrational reading level when instruction began, found 
the high-school difficulty article still at frustrational reading level at 
the end of instruction. It is unlikely that learners in either group 
received enough reading practice to bring this brief story about 
General Motors vehicles within their comprehension ranges. 

Job literacy scenarios provided a more diverse range of 
indicators of learner literacy processes and abilities across several 
types of workplace materials. Questions assessing the strategies used 
by learners in reading newsletter stories, graphs, and job aids revealed 
some change in the sophistication with which learners read. The 
Tech Prep class, which was comprised of high school graduates and 
several learners with some post high school education, scored very 
high on pre-class measures of how they went about reading. This class 
spent a good deal of time addressing study skills and reading 
strategies, and learners did score numerically higher on post- 
assessments than their already high pre-scores. Ceiling effects here 
made statistically significant improvements difficult to attain. ESL and 
GED learners demonstrated especially significant gains in topics 
focused upon. Even though they had not improved in reading abilities 
enough to do well with the earlier reported Cloze newsletter article, 
the reading practice received during their limited hours of instruction 
had led to a more sophisticated approach to reading. Control groups, 
who received no instruction, demonstrated no improvement from pre- 
to post-assessments of reading strategies. 

Comprehension questions of increasing difficulty were also 
asked in the different job literacy scenarios. Learner gains again 
reflected learner instruction. Tech Prep learners, whose extensive 
class work addressed inference and problem-solving tasks, improved 
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most on the more difficult scenario questions. The ESL learners, who 
met eight hours each week and spent some time with workplace 
materials, improved most on middle level difficulty questions. The 
GED group, who met only four hours per week and did little with 
workplace materials, demonstrated gains on only one comprehension 
question related to workplace literacy. Once more, gains appear to be 
directly related to the type and amount of instruction received by 
learners. 



Family literacy 

Some impact of instruction on home literacy has already been 
discussed in an earlier section on literacy practices. Learners in 
classes which focused upon home materials did improve home literacy 
practices; those in classes which did not focus on home materials 
made no changes. Only a relatively small number of learners had 
children and were therefore qualified to answer the family/parent 
literacy questionnaire items. No class spent direct instructional time 
on family literacy and no significant gains were noted in this area. 
Though not statistically significant, there was some indication of a few 
parents taking their own children to the library more often after 
learners from the Tech Prep class were taken to the library by their 
instructor. Similarly, a few parents reported reading to children 
slightly more often. These accounts were infrequent. It appears that 
benefits of instruction do not transfer very far beyond the focus of 
actual instructional activities. 

Productivity 

For a variety of reasons discussed in Chapters 6 and 7, few of the 
productivity assessments proved satisfactory for groups as small as the 
12-15 member classes and control groups. There is some indication, 
however, that some productivity gains were directly related to type 
and amount of instruction received by learners. The only group to 
demonstrate consistently improved supervisor ratings in workplace- 
based communication and literacy use was the Cumberland 
Communication and Collaboration class. The entire class was 
structured to address these workplace communication demands. 
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Results suggested that this focused instruction was effective in these 
areas directly related to company goals. No comparable gains were 
demonstrated with Delco supervisor ratings, though difficulty in 
obtaining acceptable ratings clouds this finding. Questionnaire items 
dealing with participation in team meetings indicate that both Delco 
and Cumberland learners who participated in class group work and 
discussions did make significant gains with participation in 
discussions during team meetings. Again the relationship between 
instruction and improved performance is fairly direct. 

Conclusions from Results 

With workplace literacy instruction, it appears that you get what 
you pay for and not much more. Classes and instructors at the two 
sites demonstrated that what you choose to spend time on in class 
matters a great deal. Statistically significant gains were made by some 
students for every segment of the evaluation model. More detailed 
analysis reveals that gains occurred, however, only in areas directly 
addressed by instruction and class activity. 

This is both good news and bad news. It is heartening to know 
that instruction works. Workplace literacy programs that focus on a 
specific goal and provide significant instruction toward that goal can 
help learners improve. If time is spent providing learners with 
feedback about their improved literacy performance and developing 
literacy habits at home and work, workers will improve their self- 
concepts about their own literacy and will read more. The bad news is 
that hopes for broad transfer from relatively brief programs (as nearly 
all workplace programs are) appear to be misplaced hopes. Whatever 
effective class activity focuses upon is the major area of gain. Even 
improvement in literacy practice appears tightly related and limited to 
classroom practice. If class time focuses only on workplace activities, 
practices appear to improve only with workplace literacy materials. 
For productivity to improve, instruction needs to focus directly on 
activities involved with production. Extra dividends of transfer to 
improved family literacy seem unlikely unless instructors also spend 
time with family literacy activities. 
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This implies some hard decision-making by instructional 
planners. The results of this study make it much more difficult to 
accept the contention that any single focus of literacy instruction will 
bring improvement in a multitude of areas. It also suggests that diffuse 
instruction, which touches lightly on many areas, will not bring about 
gains of any significance in any particular area. The GED group 
improved a little in general reading strategies and may have improved 
in taking the GED test, but there are no indications that much more 
occurred. Instructional planning did not focus on much beyond the 
GED goal. The ESL group improved in oral activities and in some 
workplace activities for which they received instruction, but not in 
areas where they received little instruction. These same patterns of 
instruction being directly and narrowly related to gains hold for the 
Tech Prep class and the Communication and Collaboration class. 

This should not be taken to mean that workplace literacy 
instruction should always focus upon a single workplace goal. It is 
likely that the most beneficial mix is instruction which expar. *s 
learner practice time beyond the classroom by improving worker 
literacy practices and beliefs at home and the workplace. Since 50 or 
even 200 hours of class time is not sufficient for many learners to 
reach their full potentials, the impact of precious class time must be, 
in part, to increase literacy practice and learner independence. If 
productivity is an issue, workplace materials and activities used in 
class should be directly related to materials and activities employed 
during production. If other goals are desirable, they must be planned 
for and it seems likely that additional learning time will also need to 
be provided. 

What Has Been Learned about How to Evaluate Programs 

One of the major goals of this study was to develop a model for 
evaluating workplace literacy programs. For the most part, the pilot 
assessments validated the utility of a broad-based conceptual 
framework of adult literacy learning in the workplace. It was possible 
and productive to note gains in areas of learner literacy beliefs, 
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practices, processes and abilities, plans, productivity and family 
activities. A good deal was also learned about the limitations and 
pitfalls of particular evaluation approaches and methods. 

limitations of Questionnaires 

Time is at a premium in workplace literacy programs. Many 
programs are only able to provide brief instruction and still others lose 
money for each hour of learner time, since learners are not producing 
a profit while in class. To the degree that checklists and 
questionnaires can be used to gather information, as opposed to 
individual interviews, a substantial time saving can be made. This pilot 
assessment used overlapping oral interview and written questionnaire 
items to test the degree to which the assessment approaches 
produced similar findings. For the most part, questionnaires, though 
time-efficient, were much less effective and accurate than even brief 
face-to-face interviews. This was especially true in the areas of 
literacy beliefs and practices. In these areas, learner responses were 
very brief on written forms - even with the more literate Tech Prep 
learners. Interviewers, however, waited until learners paused in 
speaking and then asked, "Anything else? or "Can you think of any 
other examples?" until they received the answer, "No." This produced 
a good deal more information and more accurate representations. 
Questionnaire responses in these areas probably more closely reflect 
the degree to which learners could read and wanted to write. The 
questionnaire responses tended to reinforce the interview responses, 
but questionnaire assessment was often not sensitive enough to detect 
changes ~ especially on global questions about literacy beliefs or 
practices. 

Questionnaires were effective when they could be focused. For 
example, descriptions of literacy behaviors in team meetings, listings 
of recently-read materials specific to a workplace, and descriptions of 
literacy behaviors with one's children were able to elicit information 
rapidly from learners. In cases where there was an overlap between 
interview questions and these questionnaire items (e.g. home literacy 
behaviors in the interview and family literacy behaviors in the 
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questionnaire), triangulation revealed the questionnaire items to 
reflect accurately the more extensive oral comments. 

Job Literacy Scenarios 

The custom-designed job literacy scenarios were specially 
designed to reflect workplace literacy tasks of importance at each 
worksite. They attempted also to reflect the range of reading types 
present in National Assessments of adult literacy (i.e. prose reading, 
document reading, and quantitative reading). The scenarios provided, 
as much as possible, a realistic purpose for reading and attempted to 
assess both how the learner went about reading [processes] and how 
well the learner understood and could use information from the 
reading [abilities). 

These job literacy scenarios proved to be quite productive in 
assessing improvements in the sophistication with which learners 
approached reading tasks. The initial scenarios, which were limited 
to a very few comprehension questions, were somewhat productive, 
but need to be expanded to reflect more accurately gains in several 
types of reading (searching for facts, drawing inferences, and making 
applications beyond the task at hand). Instruments in Appendix A 
have been revised to reflect these changes- 
Cloze Tests 

The Cloze test was simple to construct and relatively easy to 
administer. Instructors reported little difficulty with the tests, which 
provided a sample sentence demonstrating how to fill in blanks. With 
the Tech Prep and Cumberland classes, the material selected was well 
within initial comprehension ranges. This was not true for GED and 
ESL learners at Delco, however. The high school difficulty level story 
was beyond most learners both before and after instruction. For low 
level learners, it would be desirable to construct a second Cloze test 
using simpler workplace materials. 

Though some instructors at pilot sites were familiar with the 
Cloze test procedure, others were not. Directions for how to develop 
and interpret Cloze tests were created for instructors. These are also 
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included in Appendix C with samples of Cloze tests developed at the 
pilot sites. 

Family literacy Questions 

Both oral focus group methods and written questionnaire items 
related to family and parent literacy were piloted for this study. Both 
seemed effective in gathering information. However, workplace 
literacy classes are small and the number of learners with children at 
the pilot sites was even smaller. For most items, these small numbers 
precluded meaningful statistical analysis of workplace literacy impact 
on parent literacy practices. These measures are likely to be of more 
use for special programs which focus upon the workplace/family 
connection or for much larger groups. 

Employer-Gathered Productivity Indicators 

Though previous studies have discussed the need for assessing 
productivity impacts of workplace literacy programs, few have tried to 
do so. This pilot assessment attempted to use some of the indicators 
of productivity suggested in discussion sections of studies in the 
research literature (i.e. attendance, accident reports, useful 
productivity suggestions made by employees, etc.). The pilot revealed 
that it is possible to gather such data with a minimum of effort on the 
part of employers. It also revealed that the information is not of great 
use if sample sizes are small and time between assessments is not very 
long. If a class and control group are comprised of only 15 individuals 
each, the impact upon absences of a single individual with the flu can 
overpower all other factors. This would be less likely to occur with 
much larger groups where influences of sickness would be more likely 
to balance out. Similarly, safety is an important indicator of 
productivity and many workplace literacy programs address safety. 
Accidents among a group of 15 people during a 6 month period are 
usually rare, however, and therefore not likely to be of much use in 
determining program impact. This same pattern held for productivity 
suggestions and discipline measures as possible indicators of program 
impact. Neither employer maintained data on individual employee 
productivity, so those measures were not available. Such indicators 
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are likely to be of worth when available. 

Supervisor Ratings 

Specially constructed supervisor ratings of employee 
productivity with literacy and communication behaviors on the job 
were of greater use* Discussions with supervisors and top employees 
were able to identify the types of literacy, problem solving, and 
communication considered important on particular jobs, A careful 
process of developing and revising scales to reflect these discussions 
is available in Chapter 6 (see also Appendix G.) At the Cumberland 
plant, supervisor ratings proved to be useful in noting employee 
improvements on the job- The ratings were less successful at Delco. 

The reasons for this lack of success at Delco bear some 
examination- Delco supervisors generally had less education than the 
Cumberland supervisors and were less familiar with the concept of 
individual employee evaluation. Individual appointments to have 
researchers meet with supervisors to rate learners 1 job performance 
were cancelled for several legitimate reasons. As a result, supervisors 
sometimes rated employees without someone to remind them to think 
carefully about each scale. The resulting ratings seemed to reflect a 
desire to complete the task rapidly (e.g. many workers received 
exactly the same rating on each scale). It seems advisable in the future 
to require that supervisors make ratings with a researcher asking the 
questions and encouraging careful consideration of each scale and 
each worker. 

Supervisor ratings were possible at the two pilot sites because 
learners came from highly similar jobs. Programs that draw into 
classes learners from several different jobs may not be able to use 
supervisor ratings to assess impact on productivity. Unless jobs have 
several common tasks, it will not be possible to construct scales which 
can be used by all learners. If several different scales need to be 
constructed for several different jobs, the small number of learners in 
each Job category is likely to preclude any meaningful statistical 
analysis. 
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Learner questionnaire items related to literacy and 
communication practice on the job were of some use in gaining a 
picture of impact on productivity. These items are subject to some of 
the same limitations as supervisor ratings. At the two pilot sites, the 
expectation was for all workers to become involved in team meetings. 
For this reason, it was possible to have several questionnaire items 
related to such meetings. A workplace literacy program without such 
workplace commonalties would not be able to benefit from these 
questionnaire items. 

Conclusion 

This study has shown the feasibility of using a detailed impact 
assessment model with workplace literacy programs. Without 
requiring a large commitment of resources, it is possible to gather a 
great deal of information on learners' own literacy, the literacy of their 
families, and their job productivity. 

The results of the study indicate what can be expected of 
effective workplace literacy programs. Instruction has produced some 
improvement in all of the areas assessed, but gains appear to be 
limited to areas directly addressed in class. There is apparently no 
transfer of learning into areas not covered by instruction. Because of 
this, it appears that program providers need to have clear goals for 
what they want to achieve in the limited time that learners are in 
class. They should also be seeking ways to extend this time beyond 
the classroom. One way of doing this is to use on-the-job materials in 
class so that learners will be practicing outside class time. Also, 
encouraging learner motivation and independence is likely to lead to 
learners' engaging more often in literacy-related activities. 

The second phase of this study aims to determine whether the 
assessment model can be used by other workplace literacy programs, 
with a minimum of assistance from project personnel. In addition, 
results from this second phase will throw more light on the 
conclusions reached here in the pilot assessment. 
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Appendix A 



Interview Form 

and 

Instructions for Custom Designing 
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Interview 



What modifications are needed? 

The Interview protocol that follows addresses learners' beliefs, 
practices, processes and plans related to literacy activities. Most 
programs can use the supplied questions concerning: 

• beliefs 

• practices 

• plans 

without any modifications. 

For the process section, job-specific modifications are required 
to determine how well employees read material from a particular 
workplace. This involves selection of reading materials which are key 
to performance at that workplace. These will be used to develop three 
job reading scenarios. We recommend that you select: 

• prose material (e.g., a newsletter article) 

• a graph (e.g., a key graph or chart) 

• a procedure (e.g., an instruction sheet or job aid) 

Guidelines below provide directions for developing process, 
factual, inference and application questions for each job reading 
scenario. 
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INTERVIEW 

Personal Information: 



Name: Date:. 

What class are you in? 

Job you do 



I'd like to ask you some questions about reading, writing, and 
education. The answers to these questions will give us an idea of the 
way reading and writing are used here. 

Beliefs 

1. Describe someone you know who is good at reading and 
writing. What makes you choose this person? 



2. How good do you consider yourself to be at reading and writing? 
What makes you think so? 



3. Describe how you would like to be in terms of reading and 
writing. {Probe : Could you give me some examples?) 
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Practices 

1. Tell me the sorts of things you read and write away from work 
during a normal week. 

[For probe, ask: "Can you give me more examples?") 



2. Tell me the sorts of things you read and write on the job during 
normal week. 

[Use probe above for more examples.) 



Instructions for Custom-Designing 
Process : article, graph, and procedure/job aid 

(For Interview, following pages) 



INSTRUCTIONS 



This approach reveals far more than standardized tests about 
an interviewee's thinking processes while reading and how 
those thinking processes apply to job- related reading. 

A. You need to find materials key to your workplace for the 
scenarios. A typical mix is: 

• an article, 

• a graph, and 

• an instruction sheet. 

B. For each of these, ask 7 questions: 

• 1 process question - what is going on in the 

interviewee's mind. (Use those in our examples.) 

• 2 factual questions- based strictly on the material. 

• 2 inferences questions - deductions from the material that do 

not rely on a great deal of background knowledge. 

• 2 application questions - relating information from the 
material to the interviewee's background knowledge 
(e.g.: the employee's job). 

C. Include a range of difficulty in your questions, finishing 
with an open-ended question which allows the interviewee 
to contribute. 



Ill 




WMMMmm 
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Process : Article Example 



( 



Competitor Close Up 

1 I am going to show you a newspaper article about your industry. 
Explain to me how you would read this story in order to find out 
what the writer thinks. 

{Show attached story: "Competitor Close Up"). 
Describe what you would look at. What would you be thinking 
about? How would you go about reading this story? What would 
you do first, then next, then next? 



(easy factual question) 

How many employees does ASMO have in Statesville? 

(Answer: 400. Listed in article. ) 



(harder factual question) 

What is the only company that does not mention customers? 

(Answer: BG Automotive Motors, Inc. Requires the interviewee 
to look at all "customers" in the article. ) 



(easy inference question) 

From the information provided about products, what do all four 
companies have in common? 

(Answer: All of them make some sort of motor. Requires the 
interviewee to search for commonalities not readily apparent. ) 
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Process : Article Example (cont.) 



5. (harder inference question) 

Which of the companies listed is closely related to Japan and 
why do you think so? 

(Answer: ASMO or Jideco. Each has Japanese plants listed 
and each sells to many Japanese affiliates and main 
customers. Requires looking at two pieces of information . 
and drawing deductions based on what is provided. ) 



6. (harder application question) 

What company makes products closest to your job at this 
facility? Why do you say so? 

(Answer: Relate a product on the list to what the employee 
makes. Requires the employee to sort through the 
information and then to apply it to his/her background 
knowledge. ) 



7. (easy application question to end the section) 

From this list, which company pays the least amount to its 
workers? How does this relate to your wages at Delco? 

(Answer: ASMO. It's more or it's less than what I get paid here. 
Requires the employee to apply the information to his/her 
background knowledge, but allows him/her to contribute 
more. ) 
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Competitor Close-Up: 
A Year in Review 



Throughout the year, the Delco Doings 
has brought you profiles on the companies 
trying to take a bite out of our business and , 
our profits. Sometimes there were success 
stories, when Rochester Operations met the 
challenge and came out on top. Other times 
we had to face the fact that there are 
companies in Asia, Europe, and right here 
at home that are reaching the market better, 
faster or with lower prices. 

Here's a quick recap of the competitors 
we've covered this year. 

ASMO. Inc. 

Location: Battle Creek, Michigan; Statesville, 
North Carolina; Kosai City, Japan. 
Affiliate: Nippondeso 

Products: wiper systems, windshield washer 
systems, power window lifts, antennas, 
retractable and blower motors. 
Main Customers: Nippondeso, Ford, 
Chrysler, General Motors, and every 
Japanese transplant except Nissan. 
Number of Employees: Battle Creek, 130; 
Statesville, 400. 

Total Wage and Benefit Cost/Hour. $9.58 
Jideco 

Location: Bardstown, Kentucky;Yokohama 
City, Japan; Nine production facilities 
throughout Japan 

Affiliates: Hitachi (24%), Nissan (21%) 
Products: wipers, transmissions, reservoirs 
arms and blades, wiper motors, and others. 
Control- wiper switches and others. 
Motors -- power seat sliders, power window, 
door lock, blower and engine cooling 
motors and others. 

Accessories: air compressors, power window 

kits, door locks, rain-sensing intermittent 

wiper controls and others. 

Main Customers: Nissan, Isuzu, Honda, 

Mitsubishi, Mazda, and Suzuki. 

Number of Employees: Bardstown, 60 in 1987 

Total Wage and Benefit Cost/Hour $10.27 



Power Motion 

Location: Two plants in London, 
Ontario 

Parent: Siemans Automotive of 
West Germany 

Products: air moving motors (5,250 
armatures a day) 

Main Customer: GM of US & Canada 
Number of Employees: 200 at main 
facilitv in London, Ontario 
Total Wage and Benefit Cost/Hour 

$11.50 (U.S. equivalent) 

BG Automotive Motors. Inc. 

Location: Hendersonville, TN. 

Parents: Bosch Corporation and General 

Electric Company 

Products: 20 different small motors 

including: engine cooling, modular wipers, 

door lock, seat back, head rest, sun-roof, 

washer pump, head lamp, power window. 

Number of Employees: 275 

Total Wage and Benefit Cost/Hour 

Unknown at this time. 

Every day another company steps into 
the automotive arena ready to try to take 
away our customers. Rochester Operations 
has an extensive communication network 
to keep employees informed about our 
competitors and what we're doing to stay 
ahead. Look to Delco Doings to give you 
the information you need to help keep 
Rochester Operations competitive in the 
'90s. 



From: Delco Doings, December/ January, 1991, p.2. 
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Process : Graph Example 



Production Problems 

1. I am going to show you a graph. Explain to me how you would 
read this graph in order to find out what it's about. 
{Show attached graph." Production Problems"). 
Describe what you would look at. What would you be 
thinking about? How would you go about reading this graph? 
What would you do first, then next, then next? 



2. (easy factual question) 

What is the total number of culls? 

(Answer: 149. Shown at top of graph.) 



3. (harder factual question) 

What time period is covered in this chart? 

(Answer: one week or week one in May. Shown at top of 
graph in abbreviated form.) 



4. (easy inference) 

What is the biggest problem here? 

(Answer: tear outs. Longest bar on graph.) 
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Process : Graph Example (cont.) 

5. (harder inference) 

Find 3 types of problem involving measurement. 

(Possible answers: thickness, length, width, squareness. 
Requires selection from list at left of graph.) 



(easv application question) 

Pick one problem and suggest at least one cause for that 
problem. 

(Possible answers: For example, tear outs are caused when 
the wood gets caught in the machine and is gouged; moulder 
burn is caused by wood getting caught in the machine and 
being burned. Uses interviewee's job-related knowledge. ) 



(more difficult application question) 

Pick a second problem and suggest both a cause and a solution 
for the problem. 

(Possible answers: tear outs, caused when the wood gets 
caught in the machine and is gouged, can be repaired with 
wood filler and sanding; or moulder knife marks can be 
caused by gouging of the wood in carving it and can be 
repaired if you can get at the gouge and sand it and 
provided the finish hasn't already been applied. Uses 
interviewee's job-related knowledge in more depth.) 
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Process : Procedure /Job Aid Example 



OSHACARD 

1 The government has safely regulations and special labels in 
many workplaces. I am going to show you a safety card that 
many employees in America must keep in their pockets while 
working. This card shows how to understand safety labels. 
Explain to me how you would read this card. 
{Show attached card , "OSHA "). 

Describe what you would look at. What would you be 
thinking about? How would you go about reading this card? 
What would you do first, then next, then next? 



2 (easy factual question) 4 
What should you do when you see the letter x ? 

(Answer: Ask my supervisor. Directly explained in the text ) 



3. (harder factual question) 

What do all the symbols in M k" represent? 

(Answer: airline hood or mask, gloves, a suit and boots. 
Answers are in the text, but are wore difficult to find. ) 



4. (easy inference) 



(easy inierencej ..rrno 
What is the most common type of protection from A to k. . 

(Answer: gloves. Requires the interviewee to look through 

several parts of the text and then to generalize the information ) 
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Process : Procedure /Job Aid Example (cont.) 



5. (harder inference) 

Name all the letters which refer to severe hazards. How did 
you tell this? 

(Answer: F, H, J, K. Top of the table says "4 severe hazard; 
4 probably means 4 pictures. These letters have 4 pictures. 
Requires the interviewee to make deductions between 
different parts of the card. ) 



6. (harder application) 

If a supervisor says you are about to do a job that requires sandi 
which protective items would you choose? 

(Answer: safety glasses and a dust respirator. Optional: gloves, 
combination dust/vapor respirator and a face shield. Requires 
the interviewee to interpret the information on the card and 
to relate it to a real-life situation. ) 



7. (easy application question to end the section) 

Give me two examples of how you or someone you know 
could use this card. 

(Answer: Must give 2 examples and list protections. This 
is more open-ended and allows the interviewee to contribute 
based on his I her job background.. ) 
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Bans 

Now I'd like to ask you about your plans. Explain how you see reading 
and education as pa rt nf these plans: 

A What are your plans for the next year? 

r' 

B. What are your plans for the next 5 Years? 
C What are your plans for the next 10 Years? 
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Appendix B 



Questionnaire Form 

and 

Instructions for Custom Designing 
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Questionnaire 
What modifications are needed? 



The Questionnaire protocol that follows addresses learners' 
reading abilities, their literacy practices at work and away from work, 
and the literacy activities of their families. Most programs can use the 
supplied questions concerning: 

• literacy away from work 

• literacy at work 

• family literacy 

without any modifications. 

The section on self-rating of reading ability has 15 questions, 10 
of which should apply to most industries and thus need no changes. 
However, the last 5 items should be site-specific reading materials, 
such as warning labels, route sheets, product lists, etc. Actual names 
may differ from site to site. 

When you choose these last 5 items, select a mix of: 

• prose and graphic materials (e.g., a note from a 
supervisor, and a blueprint) 

A 

• easy and difficult reading materials (e.g., simple 
suggestion forms and more complex benefit 
information) 
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QUESTIONNAIRE 

Name: Age: Sex: 

Education: (furthest year in school) Training _ 

Marriage Status: 

Children: (number) (ages) 

Practices: Self rating reading ability 



I. 1. First check only the things you've read in the past month. 

2. Now go back and rate your ability to read the items you've checked. 

poor excellent 



local newspaDers 
classified ads 


1 a 
1 2 


o 

3 


4 5 


telephone bills 


1 2 


3 


4 5 


TV guide listings 


1 2 


3 


4 5 


magazines 


1 2 


3 


4 5 



poor excellent 

training guides 1 2 3 4 § 

paycheck stubs 12 3 4. 5 

_ company newsletter s 1 2 3 4 _5 

_ benefit information 1 2 3 4 5 

_ graphs and charts 1 2 3 4 5_ 

poor excellent 

1 2 3 4 5 

1 2 3 4 5 

12 3 4 5 

12 3 4 5 

12 3 4 5 
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Instructions for Custom-Designing 
Practices; Self rating reading ability 

(For Questionnaire, preceding page) 



INSTRUCTIONS 

1. This component includes ten standard items followed 
by five examples of site specific items. 

2. In the site specific section, replace example items 
with items key to your workplace. 

3. Have a mixture of prose and document/ chart items. 

4. These should cover a range of reading difficulties. 

5. Possible sources of materials: 

• warning labels. • production quotas. 

• parts lists. * product lists 

• department inventories. 



EXAMPLE : 

poor 

blueprints 1 

route sheets 1 

notes from supervisor 1 

suggestion forms 1 

inventory graphs 1 



excellent 



2 
2 
2 
2 
2 



3 
3 
3 
3 
3 



4 
4 
4 
4 
4 



5 
5 
5 
5 
5 
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Practices: R eading frequency 

Please check the number of times you have done the following: 

1. In the last 7 days how many times have you used a TV guide listing to 

select programs? 

0 f 2 ~3~ 4 ~5~ 6 ~7 8 9 10+ 

2. In the last 7 days how many times have you read a newpaper? 

0 f 2 ~3~ 4 ~5" 6 ~7 8 ~9 10+ 

3. In the last 7 days how many times have you read a magazine? 

0 f 2 ~3~ 4 ~5~ 6 ~7 8 9 10+ 

4. In the last 7 days how many times have you read a book for pleasure? 

0 f 2~~"~3~ 4 5~ 6 T 8 9 10+ 

5. In the last 7 days how many times have you read the following 

types of books? 

mystery: _ times how-to books: times 

novels: _ times factual books: _ times 

poetry: _ times encyclopedia: _ times 

Bible: times comic books: times 

other types: — times 

times 
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Practices: R eading frequency (cont.) 

6. How often do you make a shopping list before you go to the store? 

never occasionally often always 

7. When you're waiting in an office, how often do you read magazines? 
never occasionally often — always 

8. Do you subscribe to any magazines? yes — no 

If yes, which ones? 

9. How many different magazine titles do you have in your home? 

0 1 2 ~3 4 ~5~~'~~6 T 8 9 10+ 

10. How many books are in your home, either owned or borrowed? 

o lb 2b~'Y6~~~40~~To~~60~~To~~~86~~9b"~~roo+ 
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Practices: Literacy at work 



Please circle the number w hich best describes you in the situations below: 

(1) You just listen in team or department meeting discissions. 

very like me 1 2 3 4 5 very unlike me 

(2) You talk a lot in team or department meetings, asking questions 

or sharing ideas. 

very like me 1 2 3 4 5 very unlike me 

(3) Your ideas are often discussed in team or department meetings. 

very like me 1 2 3 4 5 very unlike me 

(4) You wait for others to talk about written information, just to be 

sure what is in it. 

very like me 1 2 3 4 5 very unlike me 

(5) You look for printed directions to help figure out what to do when 

a problem arises. 

very like me 1 2 3 4 5 very unlike me 

(6) You often have trouble reading paperwork from management. 

very like me 1 2 3 4 5 very unlike me 

(7) When the booklet about new health benefits arrived, you read it 

carefully. 

very like me 1 2 3 4 5 veiy unlike me 
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Practices : Family Literacy- 



Only answer the following questions if you have a child 
between the ages of 3-17 at home. 

Please answer for your youngest child in this age group and please fill 
in only one answer per question: 



1. This child is years old. 

2. In the last 7 days how many times has your child looked at or 

read books or magazines? 



0123456789 10+ 



3. In the last 7 days how many times has your child seen you reading 
or writing? 



0123456789 10+ 



4. In the last 7 days how many times have you helped your child 
with homework and /or with school projects? 



0123456789 10+ 



5. In the last 7 days how many times have you read /looked at books 
with your child or listened to him/her read? 



0123456789 10+ 



6. In the last 7 days how many times has your child asked to be 
read to? 



0123456789 10+ 



7. In the last 7 days how many times has your child printed, made 
letters, or written? 



0123456789 10+ 
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Practices : Family Literacy (cont.) 



8. In the last month how many times has your child gone to a public 

library? 

0 f 2 ~3"~~4~~~~5~~~~6 7~ 8 9 10+ 

9. In the last month how many times have you participated /helped 

out in your child's school? 



0 f 23456789 10+ 

10. In the last month how many times have you hung up or displayed 
your child's reading and writing efforts? 



0 



8 9 10+ 



11. In the last month how many times have you bought or borrowed 
books for your child? 



0 f 2~ 3 4 5 6 7 8 9 10+ 



12. (Please check only one.) 

I expect my child to finish at least: 

_6th grade _9th grade _high school 

_two-year college _4-year college or more 
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Appendix C 



Cloze Test Samples 

and 

Instructions for Custom Designing 



Cloze Exercise 



The cloze procedure is based on the psychological principle of 
closure, which is the human tendency to recognize and complete a 
pattern or sequence. It involves replacing missing words in a reading 
passage. This procedure can assess the ability of employees to 
comprehend the passage. Cloze test scores correlate very highly with 
standardized reading test scores. Cloze tests can be made from local 
workplace materials. 



CLOZE PROCEDURE 

1 . Select a job relevant passage of 150-200 words. 

2. Leave the first and last sentences intact. 

3. Starting with the second sentence, omit every fifth 
word. This will give about 25 blanks. Replace all 
omitted words with the same sized blank line, 
13-15 spaces is typical. 

4. Employees are to read the passage and to fill the 
blanks with their best guess at the word removed. 

5. Instructions for the cloze test should suggest 
employees read the entire text before attempting 
to fill in the blanks and should encourage 
employees to answer all questions even if they 

have to guess. It is rare for anyone to know more 
than 50% of the blanks. 

6. Avoid controversial or emotional topics, and 
topics requiring technical knowledge. Scores for 
such materials are less valid. 



144 



Name Date 

CLOZE Exercise 

In a cloze exercise, you try to guess which words are missing. For 
example, in the sentence below, a word is missing. 

She looked before she the street. 

A good guess for the missing word is "crossed." 

She looked before she crossed the street. 

In the story below, try to guess and replace the missing words. Don't 
expect to get them all. Some are nearly impossible. 

GJVI Designs Safety for All Ages 
We all like to think about the old days. Life seemed simpler and, in 

some ways, better then. But when it comes to , the 

good old days offer the same degree 

safety as today's cars trucks. Advancements in 

technology the G.M. vehicle you 



today among the safest the world. Each G.M. 

and truck is backed . thousands of 

dedicated men women who care about 

safety of their customers. . as G.M. 

customers themselves, have a stake in 

G.M. vehicles the highest quality 

and reliability. 

And you're wondering if safety 



improved in recent years. this: The classic 1955 

would require more than major 

changes or additions hundreds of incremental 

changes be as safe as vehicles. 

From: Kilbom. C. OM Today (November/December, 1990), page 1. 
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Cloze Exercise Answer Key 

q.M Designs Safety for All Ages 
We all like to think about the old days. Life seemed simpler and, in 

some ways, better then. But when it comes to automoSiCes , the 

good old days didn't offer the same degree of safety 

as today's cars and trucks. Advancements in 
technology make the G.M. vehicle you purchase 

today among the safest in the world. Each G.M. 

car and truck is backed Su thousands of 
dedicated men and women who care about 

the safety of their customers. And , as G.M. 
customers themselves, theu have a stake in 

making G.M. vehicles the highest in quality and 
reliability. 

And if you're wondering if safety has 

improved in recent years, consider this: The classic 1955 

Chevrolet would require more than 60 major 

changes or additions and hundreds of incremental 
changes to be as safe as today's vehicles. 
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Name or ID# Date 

CLOZE Exercise 

In a cloze exercise, you try to guess which words are missing. For 
example, in the sentence below, a word is missing. 

She looked before she the street. 

A good guess for the missing word is "crossed." 

She looked before she crossed the street. 

In the story below, try to guess and replace the missing words. Don't 
expect to get them all. Some are nearly impossible. 

Two more teams on the self-d irected journey 

Our workplace is taking on more change daily. So are the skills 
that all our employees must have in order to change with it. It is 

getting to be essential each day that skill 

gaps be filled our small business can a 

source of competitive • 

The changes all companies expect over the next 

— a shrinking labor force, demand for 



workers in jobs, and increasingly competitive 

markets -- will require businesses all sizes 

to strengthen employee skills and training 



We believe that our firm can remain competitive 

the large firm by a more flexible training 

education program. We hope be better than the 

firm in adapting an previous training 



experiences to company's needs. 

Two more are now involved in self-directed 

team training. They are Green Team and the 

Team. They join the Orange Team, which completed their sessions 
last year. 

147 14 '< 



Cloze Exercise Answer Key 
Two more teams on the self-directed journey 



Our workplace is taking on more change daily. So are Hie skills 
that all our employees must have in order to change with it. It is 

getting to be more essential each day that the 

skill gaps be filled SO our small business can remain 

a source of competitive Strength . 

The changes all companies Can expect over the next 
decade - a shrinking labor force. more demand for workers 
in technical jobs, and increasingly competitive world markets - 

- will require businesses Of all sizes to strengthen their 

employee skills and training programs . 

We believe that our small firm can remain competitive with 
the large firm by having a more flexible training — and 
education program. We hope to be better than the large 
firm in adapting an employee's previous training experiences to 
the company's needs. 

Two more teams are now involved in self-directed — WQTk 

team training. They are the Green Team and the White 

Team. They join the Orange Team, which completed their sessions 
last year. 
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Name or ID# 



Date 



CLOZE Exercise 

In a cloze exercise, you try to guess which words are missing. For 
example, in the sentence below, a word is missing. 

She looked before she . the street. 

A good guess for the missing word is "crossed/' 

She looked before she crossed the street. 

In the writing below, try to guess and replace the missing words. 
Don ? t expect to get them all. Some are nearly impossible. 

Cumberland Safety EuLes 

1. For your welfare, all injuries, no matter how slight, incurred on Company premises 

must be reported immediately to your supervisor. The services of a physician 

available and will be as required. Failure to 

such injuries may cause difficulties and could affect 



2. Wear 



Open toed or 



Workingmen's Compensation benefits. 

and shoes suitable to 

top shoes are not 



work. 



are required. 

a. Dust your only with an air 



Shorts are not permitted. 



equipped with an air 



nozzle. 



4> Keep fire equipment 
rules. 

Sw Learn 

your 



its proper place and 



all fire 



lift properly. Keep your 



^straight and use 



a All 



to avoid strain, 
guards should be kept 



machine guards 

guurd 

7. Do 



be reported to your 



place. Unsafe 
at once. No 



be removed without the 



repair machinery when it 



of your supervisor. 
in operation. 



Stop it and fix the switch so that it cannot be accidently turned on. 
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Cloze Exercise Answer Key 
Cumberland Safety Rule s 



1. For your welfare, all injuries, no matter how slight, incurred on Company premises 
must be reported immediately to your supervisor. The services of a physician 

are available and will be obtained as required. Failure to 
report such injuries may cause medical difficulties and could affect 
your Workingmen's Compensation benefits. 

2. Wear clothing and shoes suitable to _ygur_ work- 
Open toed or _caim£- top arp nnt permitted . Shorts are not permitted. 

Shirts are required. 

3. Dust your clothes only with an air hose equipped with an air 

restricting nozzle. 

4. Keep fire equipment in is proper place and obey all fire 
rules. 

5. Learn to lift properly. Keep you r back straight and use 
vou r leas to avoid strain. 

6. All machine guards should be kept ZQ_ place. Unsafe 
machine guards should be reported to your _ sup?rvi$<?r at once. No 
guard should be removed without the permission of your supervisor. 

7. Do not repair machinery when it is i n operation. 

Stop it and fix the switch so that it cannot be accidently turned on. 
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Appendix D 



Family Literacy 
Focus Group Interview 



1 A B 

JL i w 
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Family literacy Focus Group Interview 



This interview form is designed to be used with a group of 
learners as the basis for a discussion about family literacy. It has been 
found that the comments of one member of the group will stimulate 
the thoughts of others, producing a wider range of ideas than will 
individual interviews. 



1 
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Family literacy Focus Group Interview 



Why do you think some children learn to read and write well 
in school and others don't? 



Do you think there is anything parents might do to help their 
children learn to read and write better? 



Do you keep any reading or writing materials at home? 
(i e . letter blocks, flashcards, paper, pens, chalkboard, books, 
magazines, comics, cassettes with books, encyclopedia, 
dictionary, newspapers, etc) 
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4. Do you do any reading or writing activities with your children? 
(i.e., visit library, hear stories, read to them, watch educational 
television, look at magazines or books with children, point out 
words to them, play school, show them how to read or write, 
etc) 



5. At home, do your children see you doing any reading or 

writing? (i.e. books, magazines, papers, recipes, directions, 
letters, lists, notes, etc.) 



6. What activities are you involved in at your child's school? 
(i.e., parent/teacher meetings, school fund-raisers, 
committees, notes or letters, informal talks when collecting 
child, assist in classroom, help child read at home, etc) 



ERIC 
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7. 



Have you begun anything new related to reading and 
writing since you started classes here? 



a. Materials 



b. Activities 



c. Modelling 



d. School 
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Appendix E 



Classroom 
Observation Form 
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Classroom Observation Form 



The classroom observation form serves as a guide for recording 
notes about the activities actually occurring in the classroom. It is 
divided into columns reflecting the time in five minute intervals, the 
actual activities of both teacher and student, and comments about the 
overall class. The form suggests items the observer might wish to 
note. 



Classroom Observation 



Time 

0 



05 



10 



15 



20 



25 



30 



35 



40 



45 



50 



55 



Student Activity 



Teacher Activity 



Comments 



Make note of time spent by students actually reading or doing things. Also note time 
learners spend listening to the instructor. When learners are in small groups or 
working individually should be mentioned. Special note should be made when the 
instructor or a student demonstrates how-to do something. 
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Appendix F 



ESL Checklist 
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ESL Checklist 



The ESL checklist is designed for teachers to reflect upon the level 
of competence each student is demonstrating. Teachers will be able 
to note individual areas of strength and weakness. This form is helpful 
both in planning instruction and in suggesting areas for the student to 
practice on outside of the workplace. 



ESL Benchmarks and Ratings* 
Learner Name Teacher Name 



Date of Rating 

For each item, rate the learner: 

3 ss can do this as well or nearly as well as a native speaker 
2 = can usually manage to do this, but sometimes has trouble 
1 = can only sometimes manage to do this adequately 
0 = cannot do this 

Be ginner Level 

Briefly describes feelings about work 

Briefly describes feelings about other life areas 

Follows simple directions 

Asks for clarification if something is not understood 

Reads alphabet in English 

Word recognition: 

has access to dictionary/understands dictionary use 

uses dictionary 

uses roots, prefix, suffix 

4 

uses context 



Looks up simple information (phone book, dictionary) 

Reads simple signs 

Begins short journal entries 



* Modified from Bronstein, E. (1991) Benchmarks and student learning profile for the 
workplace ESL program of the Labor Education Center at Southeastern Mass. 
University 
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Intermediate Level 
Oral 

Discusses feelings about work with some elaboration 

Discusses feelings about other life areas with some elaboration 

Gives /follows instructions at work 

Gives /follows instructions in other life areas 

Asks for clarification if something is not understood 

Discusses industrial specific diseases/illnesses 

Describes /reports dangerous conditions 

Offers suggestions to supervisor 



Reading 

Uses dictionary ( bilingual English-English) 

Locates own reading material in newspapers 

Understands literal level of text 

Infers information not explicitly stated 

Draws conclusions from reading 

Writing 

Fills out more complex forms 

job application social security insurance 

other application forms (library card, courtesy card, credit 

card) 

Writes short notes /memos ( at work 

out of work) 

Writes journal entries (dialogue journal) 

Uses correct punctuation 
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Advanced Level 
Oral 

Discusses feelings with more elaboration 

Asks for clarification if something is not understood 



Gives /follows more complex directions 
Understands and can discuss basic worker rights 



Understands and can discuss key contract sections 



Reading 

Uses index/ table of contents 



Locates own reading material in newspaper 

Locates own reading material in encyclopedia or other 
reference 

Reads short pieces in newspaper and simple magazines 
Reads short pieces in flyers, notices, factsheets 
Reads short self-selected material at home 



Reads longer pieces in books or longer articles 



Writing 

Fills out more complex forms 

job application social security insurant 

other application forms (library card, courtesy card, credit 

card) 

Writes short notes/memos ( at work 

out of work) 



Observes differences in tone /register between formal and 
informal writing 

Writes longer journal entries /responds to entries 
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Appendix G 



Supervisor Rating Scales 
Examples and Instructions 
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Developing Supervisor Ratings 

It is best to develop ratings of employee job performance 
together with supervisors and possibly key employees. 

1. First ask supervisors to describe how top performers use s 

information on the job. 

Encourage them to think of specific workers who are top 
performers. A supervisor might say. for example, that a top 
performer reads charts and responds with his own analysis, 
or sets machines correctly and checks settings thoroughly, 
or completes all job-related paperwork and tries to improve 
procedures. Continue to probe until you feel reasonably 
satisfied you have a complete list. From this list, you can 
identify important areas (i.e. communication, problem 
solving, paperwork, etc.) 

Next ask supervisors to go through a three-step process in 
fleshing out these areas. The order of these steps is important. 

2. Ask supervisors to: 

a. describe the behavior of the top performers first; 

b. then, describe the behavior of the bottom performers; 

c. last, describe the average performer. 

These behaviors will be used to provide descriptions and 
anchors for ratings. In relation to paperwork, for example, 
supervisors might agree on the following descriptions: 

Top: completes all job-related paperwork and tries to 
improve procedures; 

Jottom: intimidated by job-related paperwork and does it 
poorly; 

Average: does job-related paperwork but simply keeps pace. 
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As supervisors develop these descriptions, new areas and 
categories may emerge. The supervisors may give examples 
related to problem-solving or to machine setting, or some 
other area. These may later become additional rating 
scales. 

3. Once the descriptions of top, bottom, and average performances are 
completed, work with supervisors to develop acceptable labels 

for the categories. 

For example, labels might include items like machine 
setting, paperwork, communication, responsibility, and 
problem-solving. 

♦ 

4. After this discussion, you will draft a rating scale and submit it to 

the supervisors for comment and possible revision. Sometimes 
during revision, complex scales split to become two separate 
scales. 

Examples of scales appear on the following pages. 
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Employee Assessment - Overall Rating 



Please rate each employee on a scale of 1 - 10 for each aspect below. 

• An average employee would be rated 5. 

• A top employee would be rated 8 or higher. 

• A bottom employee would be rated 2 or lower. 



EMPLOYEE 



DATE 



RATER 



COMMUNICATION 

Bottom 

won't speak; 
can't express self; 
nervous; won't 
shake hands 



Average 

open, relaxed 
communicator; 
good listener 
and responder 



Top 

processes 
information 
and responds 
with own analysis 



123456789 10 



CONCERNS. PROBLEM-SOLVING 
Bottom Average 



doesn't consider 
alternative solutions; 
makes irrelevant 
suggestions; 
never thinks of 
consequences 



can suggest 
solutions, but 
not work through 
them in detail 



To£ 

suggests solutions 
and analyses 
consequences, 
including a 
timeline 



8 



10 



9 

ERIC 
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HANDLING CONFLICT 



Bottom 



Average 



Top 



antagonistic; 
turns back on others; 
makes abrupt denials 
and impolite comments 



cooperates with 
others most of 
the time, but some 
antagonism 



empathetic; 
cooperative; 
consistent 
attitude 



8 



10 



SELF-ESTEEM 
Bottom 

shy; uncertain; 
overwhelmed by 
life's problems 



Average 

some confidence 
in self; but life 
not really 
under control 



Top 

confident; 
usually in control 
of life and of 
most situations 



- 4 5 6 7 8 9 10 



SETTING GOALS 

Bottom 

unable to plan 
ahead and 
set goals 



Average 

some short-term 
planning and 
goal setting 



Toe 

clear plans for 
future; definite, 
reachable goals 



______ 5 6 7 8 9 10 
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COMMITMENT 



Bottom 



Average 



Top 



lacks motivation; 
no interest in 
company goals 



some commitment; 
but just doing a 
competent job 



conscientious; 
committed to 
company goals 



--------- 3456789 10 



RESPONSIBILITY 
Bottom 

has to be told what 
to do and checked on 



Average 

can be left to cany 
out routine work 



Top 

dependable; takes 
responsibility 
for own work 



12 3456789 10 



INITIATIVE 
Bottom 

ignores machine 
errors and lets 
them build up 



A* gagg 

realizes machine 
errors and 
attempts 
immediate 
solution only 



Toe 

monitors machine 
errors and deals 
with them 
through the team 



l" 2 " "3 " 4 5 6 7 8 9 10 
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PAPERWORK 



Bottom 

intimidated 
by job -rented 
paperwork and 
does it poorly 



Average 

does job -related 

paperwork, 

simply 

keeping pace 



Toe 

completes all 
job-related 
paperwork and 
tries to improve 
procedures 



l" "~2~ 3 4 5 6 7 8 9 10 



MACHINE SETTING 
Bottom 

unable to set 
machines correctly 



Average 

usually sets 
machines correctly, 
but doesn't always 
check settings 



Top 

sets machines 
correctly and 
checks settings 
thoroughly 



4 5 6 7 8 9 10 
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Tabular Data 
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Glossary of Variables 

In the tables that follow, the variables are given brief descriptions 
which may not always be entirely clear. So this glossary provides a fuller 
explanation for those variables that require it. 



BELIEFS AND PLANS 

Literacy self-rating 
Change in literacy 
self-image 

Change in plans for 

1, 5, 10 years 
Change in plans for 

education 

PRACTICES 

Reading/writing 
away from work 

Reading/writing 
at work 

Items read (in 
20 item list) 

Frequency of 
reading activities 

Ownership of 
reading materials 

Self-rating on 
talking in meetings 
Self-rating on 
ideas discussed 
Self-rating on 
asking for help 

PROCESSES 

Total process 

responses 
Focus 

responses 
Strategy 

responses 
Topic 

responses 



Learner self-rating of literacy level (on scale 1-5) 
Holistic judgement of learner's change in literacy 
self-image (on scale -1, 0, +1) 

Holistic judgement of learner's change in plans for 

1, 5, 10 years (on scale -1, 0, +1) 
Holistic judgement of learner's change in plans for 

reading and education (on scale -1, 0, +1) 

Count of types of reading/writing away from work 
in last week 

Count of types of reading/writing at work 
in last week 

Count of items from given list read in last month 

Sum of 6 frequency ratings of literacy activities 
(each on scale 1 = never to 5 = everyday) 

Sum of book ownership and magazine subscription 
( on scale 1 = 1-5 to 8 = 50+) 

Learner self-rating on their participation in meetings 
(on scale 1-5) , 

Learner self-rating on how much their ideas are 
discussed in meetings (on scale 1-5) 

Learner self-rating on how much they ask for help 
at work (on scale 1-5) 

Count of all responses to process question 

Count of responses to process question involving 

points of focus (e.g. title, bold print) 
Count of responses to process question involving 

reading strategies (e.g. skim, read through) 
Count of responses to process question involving 

topics of interest (e.g. products, wages) 



• In each pair of values, mean is above and standard deviation is below 
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Delco: Technical Preparation Class (n = 14) 



BELIEFS & PLANS 



Pre -test Post-test 



Change 
mean/s.d. 



Significance 



Literacy 
self-rating 


3.357 
1.082 


3.929 
.829 


.571 
.938 


p<.05 


Change in literacy 
self-image 


(Holistic 




0.5 
.65 


p<.01 


Change in plans 
for 1 year 

Change in plans 
for 5 years 


judgements 
of change: 
no pre- 


0.429 
.646 

0.357 
.633 


p<.05 
p<.05 


Change in plans 
for 10 years 

Change in plans 
for education 


and post- 
scores) 




0.0 
.679 

- .143 
.949 


n.s. 
n.s. 


PRACTICES 


Reading/writing 
away from work 


4.786 
2.082 


6.571 
2.709 


1.786 
2.86 


p<.05 


Reading/writing 
at work 


2.846 
1.725 


2.615 
2.293 


- .231 
3.516 


n.s. 


Items read (in 
20 item list) 


17.308 
4.644 


18.846 
2.734 


1.538 
3.799 


n.s. 


Frequency of 
reading activities 


16.308 
3.093 


16.923 
3.201 


.615 
1.502 


n.s. 


Ownership of 
reading materials 


5.154 
2.035 


5.154 
1.951 


0.0 
1.354 


n.s. 


Self-rating on 
talking in meetings 


2.769 
1.301 


3.231 
1.235 


.462 
.776 


p<.05 


Self-rating on 
ideas discussed 


2.385 
1.325 


3.231 
1.092 


.846 
1.463 


p<.05 


Self-rating on 
asking for help 


1.615 
.768 


1.615 
.768 


0.0 
1.08 


n.s. 



In each oalr of values, mean is above and standard deviation is below. 
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Delco: Technical Preparation Class (cont.) 



PROCESSES 


Pre-test 
mean/s.d.* 


Post-test 
mean/s.d. 


Change 
mean/s.d. 


Significance 


Total process 
responses 


4.214 
1.424 


5.286 
1.858 


1.071 
2.526 


n.s. 


Focus 
responses 


1.786 
1.188 


1.786 
1.251 


0.0 
.961 


n.s. 


Strategy 
responses 


1.143 
.864 


1.357 
.842 


.214 
1.051 


n.s. 


Topic 
responses 


1.28b 
1.437 


1.748 


2.143 


11. 0. 


Article question 
(easy factual) 


.929 
.267 


1.0 
0.0 


.071 
.267 


n.s. 


Article question 
(harder factual) 


1.5 
.519 


1.714 
.469 


.214 
.426 


p<.05 


Graph question 
(easy factual) 


4.857 
.363 


4.857 
.363 


0.0 
.392 


n.s. 


Graph question 
(easy factual) 


4.571 
.756 


4.643 
.633 


.071 
.917 


n.s. 


Graph question 
(harder factual) 


4.071 
1.207 


3.714 
1.139 


- .357 
1.008 


n.s. 


Graph question 
(inference) 


2.071 
1.072 


2.786 
.893 


.714 
.994 


p<.01 


Job aid question 
(easy factual) 


.929 
.267 


1.0 
0.0 


.071 
.267 


n.s. 


UUu cllU i4Ll.C0LJ.UIl 

(harder factual) 


2.429 
.646 


2.643 
.497 


.214 
.893 


n.s. 


Job aid question 
(inference) 


2.643 
.745 


2.857 
.535 


.214 
.975 


n.s. 


Cloze test score 


10.857 
2.685 


12.429 
3.131 


1.571 
2.377 


p<.05 


• In each pair of values mean is above and standard deviation is below. 
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Delco: Technical Preparation Control (n = 12) 



BELIEFS & PLANS Pre -test Post-test Change Significance 
mean/s.d.* mean/s.d. mean/s.d. 



Literacy 
self-rating 


3.583 
.669 


3.5 
.674 


- .083 
.515 


n.s. 


Change in literacy 
self-image 


(Holistic 




.083 
.515 


n.s. 


Change in plans 
for 1 year 

Change in plans 
for 5 years 


judgements 
of change: 
no pre- 




- .167 
.718 

- .333 
.492 


n.s. 
n.s. 


Change in plans 
for 10 years 

Change in plans 
for education 


and post- 
scores) 




- .083 
.515 

2.25 
.866 


n.s. 
n.s. 


PRACTICES 


Reading/writing 
away from work 


4.75 
1.42 


4.583 
1.564 


- .167 
2.29 


n.s. 


Reading/writing 
at work 


3.0 
.853 


3.083 
1.832 


.083 
1.505 


n.s. 


Items read (in 
20 item list) 


18.75 
2.34 


18.583 
2.644 


- .167 
3.81 


n.s. 


Frequency of 
reading activities 


18.167 
2.082 


16.583 
1.676 


- 1.583 
2.644 


n.s. 


Ownership of 
reading materials 


4.917 
2.019 


4.833 
2.038 


- .083 
1.165 


n.s. 


Self-rating on 
talking in meetings 


3.333 
1.614 


3.083 
1.311 


- .25 
2.094 


n.s. 


Self-rating on 
ideas discussed 


3.5 
1.314 


3.25 
1.055 


- .25 
1.138 


n.s. 


Self-rating on 
asking for help 


1.667 
1.231 


2.25 
1.485 


.583 
1.379 


n.s. 


• In each pair of values, mean is above and standard deviation is below. 
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Delco: Technical Preparation Control (cont.) 



PROCESSES Pre-test Post-test Change Significance 



Total process 
responses 

Focus 
responses 

Strategy 
responses 

Topic 
responses 



Article question 
(easy factual) 

Article question 
(harder factual) 



Graph question 
(easy factual) 

Graph question 
(easy factual) 

Graph question 
(harder factual) 

Graph question 
(inference) 



Job aid question 
(easy factual) 

Job aid question 
(harder factual) 

Job aid question 
(inference) 



Cloze test score 



5.75 


5.25 


- .5 


n.s. 


1.765 


1.545 


2.393 




1 Q1 7 


z.uoo 


167 


n.s. 


.9 


.9 


.937 




1.583 


1.0 


- .583 


n.s. 


1.443 


.603 


1.676 




2.25 


2.167 


- .083 


n.s. 


1 710 


l ftm 


1 676 




l.U 




O 0 


n s 


0.0 


0.0 


0.0 






X . JOJ 


167 


n.s. 


.515 


.515 


.718 




4.583 


4.333 


- .25 


n.s. 


.515 


.651 


.754 




O, 1 / 


^ Ql 7 


0 0 


n.s. 


1.311 


.793 


1.044 




Z. / o 


O.OJJ 


1 083 


p<.05 


1.865 


.835 


1.782 




2.75 


2.583 


- .167 


n.s. 


1.138 


.669 


1.267 




1.0 


1.0 


0.0 


n.s. 


0.0 


0.0 


0.0 




2.917 


2.833 


- .083 


n.s. 


.289 


.577 


.289 




2.583 


2.667 


.083 


n.s. 


.793 


.778 


.289 




9.583 


10.417 


.833 


n.s. 


2.843 


2.968 


1.801 





• In each pair of values, mean is above and standard deviation is below. 
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Delco: GED Class (n = 15) 



BELIEFS & PLANS Pre-test Post-test Change Significance 
mean/s.d.* mean/s.d. mean/s.d. 



Literacy 
self-rating 


2.9 
.568 


2.8 
.632 


- .1 
.568 


n.s. 


Change in literacy 
self-image 


(Holistic 




0.133 
.64 


n.s. 


Change in plans 
for 1 year 

Change in plans 
for 5 years 


judgements 
of change: 
no pre- 




0.0 
.845 

0.2 
.561 


n.s. 
n.s. 


Change in plans 
for 10 years 

Change in plans 
for education 


and post- 
scores) 




0,067 
.704 

0.067 
.704 


n.s. 
n.s. 


PRACTICES 


Reading/writing 
away from work 


3.867 
1.407 


4.2 
1.424 


.333 
1.447 


n.s. 


Reading/writing 
at work 


2.133 
1.598 


2.267 
1.71 


.133 
1.642 


n.s. 


Items read (in 
20 item list) 


18.867 
1.807 


19.6 
1.056 


.733 
1.1 

4 


p<.05 


Frequency of 
reading activities 


17.0 
2.746 


17.357 
2.56 


.357 
2.62 


n.s. 


Ownership of 
reading materials 


5.463 
2.634 


5.692 
2.136 


.231 
1.691 


n.s. 


Self-rating on 
talking in meetings 


3.0 
1.464 


3.267 
1.58 


.267 
1.58 


n.s. 


Self-raing on 
ideas discussed 


3.067 
1.28 


3.4 
1.242 


.333 
.9 


n.s. 


Self-rating on 
asking for help 


1.533 
1.125 


1.667 
1.113 


.133 
1.598 


n.s. 



• In each pair of values, mean is above and standard deviation is below. 
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Delco: GED Class (cont.) 



PROCESSES 


Pre-test 
mean/s.d.* 


Post-test 
mean/s.d. 


Change 
mean/s.d. 


Significance 


Total process 
responses 


3.333 
1.175 


5.133 
2.1 


1.8 
1.821 


p<.001 


Focus 
responses 


1.533 
.915 


1.6 
.737 


.067 
.594 


n.s. 


Strategy 
responses 


1.667 
1.047 


1.133 
.743 


- .533 
1.125 


p<.05 


Topic 
responses 


. loo 
.352 


2 4 
2.261 


2 267 
2.187 


p<.001 


Article question 
(easy factual) 


.933 
.258 


.867 
.352 


- .067 
.258 


n.s. 


Article question 
(harder factual) 


1.467 
.516 


1.267 
.594 


- .2 
.676 


n.s. 


Graph question 
(easy factual) 


3.8 
1.082 


3.933 
.594 


.133 
1.187 


n.s. 


Graph question 
(easy factual) 


3.4 
1.183 


3.467 
.64 


.067 
1.223 


n.s. 


Graph question 
(harder factual) 


3.133 
1.407 


2.667 
1.496 


- .467 
1.356 


n.s. 


Graph question 
(inference) 


1.6 
1.454 


2.067 
1.1 


.467 
1.598 


n.s. 


Job aid question 
(easy factual) 


.533 
.516 


.467 
.516 


- .067 
.594 


n.s. 


Job aid question 
(harder factual) 


2.4 
.828 


2.867 
.352 


.467 
.915 


p<.05 


Job aid question 
(inference) 


2.667 
.724 


2.4 
1.121 


- .267 
1.438 


n.s. 


Cloze test score 


7.467 
1.642 


7.933 
2.187 


.467 
2.232 


n.s. 
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In each pair of values, mean is above and standard deviation is below. 
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Delco: ESL Class (n = 15) 



BELIEFS & PLANS Pre -test Post-test Change Significance 
mean/s.d.* mean/s.d. mean/s.d. 



Literacy 
self-rating 


3.091 
.302 


3.455 
.522 


.364 
.505 


n.s. 


Change in literacy 
self-image 


(Holistic 




0.067 
.799 


n.s. 


Change in plans 
for 1 year 

Change in plans 
for 5 years 


judgements 
of change: 
no pre- 




0.2 
.775 

0.2 
.676 


n.s. 
n.s. 


Change in plans 
for 10 years 

Change in plans 
for education 


and post- 
scores) 




0.067 
.704 

0.533 
.64 


n.s. 

p<.005 


PRACTICES 


Reading/writing 
away from work 


4.8 
1.897 


4.6 
1.805 


- .2 
2.077 


n.s. 


Reading/writing 
at work 


1.867 
1.506 


2.6 
1.682 


.733 
1.624 


n.s. 


Items read (in 
20 item list) 


16.286 
3.539 


19.143 
1.657 


2.857 
3.009 


p<.005 


Frequency of 
reading activities 


15.6 
4.205 


16.333 
4.624 


.733 
1.907 


n.s. 


Ownership of 
reading materials 


4.462 
2.295 


4.923 
2.326 


.462 
1.506 


n.s. 


Self-rating on 
talking in meetings 


2.333 
1.175 


2.6 
1.242 


.267 
.799 


n.s. 


Self-rating on 
ideas discussed 


2.5 
1.401 


2.571 
1.016 


.071 
1.141 


n.s. 


Self-rating on 
asking for help 


1.267 

.594 , 

r 


1.933 
1.033 


.667 
1.234 


p<.05 


• In each pair of values, mean is above and standard deviation is below. 
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Delco: ESL Class (cont.) 



PROCESSES 



Pre -test Post-test 



Change 
mean/s.d. 



Significance 



Total process 
responses 

Focus 
responses 

Strategy- 
responses 

Topic 
responses 



Article question 
(easy factual) 

Article question 
(harder factual) 



3.533 


5.0 


1.467 


p<.0005 


.99 


1.414 


1.356 




1.733 


1.667 


- .Uo/ 


n.s. 


.884 


.724 


.961 




1.533 


1.667 


.133 


n.s. 


1.125 


1.047 


1.685 




.267 


1.667 


1.4 


p<.005 


.594 


1.543 


1.765 




.867 


.867 


0.0 


n.s. 


.352 


.352 


.378 




.8 


1.333 


.533 


p<.05 


.676 


.617 


.99 





Graph question 
(easy factual) 

Graph question 
(easy factual) 

Graph question 
(harder factual) 

Graph question 
(inference) 



Job aid question 
(easy factual) 

Job aid question 
(harder factual) 

Job aid question 
(inference) 



Cloze test score 



3.733 


3.733 


0.0 


n.s. 


1.1 


1.1 


1.363 




3.0 


3.467 


.467 


n.s. 


1.195 


1.125 


1.642 




2.4 


3.4 


1.0 


p<.05 


1.844 


1.183 


1.813 




1.733 


2.333 


.6 


n.s. 


1.335 


1.345 


1.454 




.667 


.733 


.067 


n.s. 


.488 


.458 


.458 




2.2 


2.533 


.333 


n.s. 


1.014 


.99 


.9 




1.6 


2.533 


.933 


p<.005 


1.352 


1.06 


1.163 




6.467 


7.0 


.533 


n.s. 


2.615 


3.229 


2.642 





In each pair of values, mean is above and standard deviation is below. 
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Cumberland: Communications and Collaboration Class (n = 21) 



PRACTICES ~ Pre-test Post-test Change Significance 
mean/s.d.* mean/s.d. me an/s.d. 

Self-rating on 
talking in meetings 

Self-rating on 
ideas discussed 

Self-rating on 
asking for help 



3.2 


2.933 


- .267 


n.s. 


1.424 


1.335 


1.033 




3.0 


3.333 


.333 


n.s. 


1.464 


1.345 


1.397 




1.214 


1.643 


.429 


p<.05 


.579 


1.008 


.646 





SUPERVISOR RATINGS 



Communication 


2.762 
.944 


5.048 
1.024 


2.286 
.644 


p<.0001 


Concerns, 
problem-solving 


2.667 
.796 


4.857 
1.014 


2.19 
.928 


p<.0001 


Handling 
conflict 


3.0 
1.049 


5.143 
1.315 


2.143 
1.236 


p<.0001 


Self-esteem 


2.905 
.944 


5.333 
1.39 


2.429 
1.076 


p<.0001 


Setting goals 


2.857 
.727 


4.857 
1.315 


2.0 
1.225 


p<.0001 


Commitment 


3.143 
1.276 


4.857 
1.315 


1.714 
1.007 


p<.0001 


Responsibility 


3.19 
1.169 


5.19 
1.327 


2.0 
1.095 


p<.0001 


Initiative 


3.143 
1.195 


4.714 
1.347 


1.571 
.811 


p<.0001 


Paper work 


2.429 
.87 


4.714 
1.707 


2.286 
1.309 


p<.0001 


Machine setting 


3.238 
1.179 


5.095 
1.546 


1.857 
1.276 


p<.0001 



• In each pair of values, mean Is above and standard deviation 
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